System for determining the quality of query suggestions using a network of users and advertisers

ABSTRACT

A system is described for determining the quality of query suggestions using a network of users and advertisers. The system may include a memory and a processor. The memory may store a historical dataset, a residual value, a query-advertisement link value, a query suggestion value, and a data representing a network. The network may comprise a plurality of query items linked to a plurality of advertisement items via a plurality of query-advertisement link items. The processor may generate data representing the network and may identify a query-advertisement link item in the network. The processor may calculate the residual value of the query suggestion system represented by the match type of the query-advertisement link item. The processor may calculate the query-advertisement link value. The processor may add the residual value to the query-advertisement link value to determine a query suggestion value and may store the query suggestion value in the memory.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 11/786,326, filed Apr. 10, 2007 (pending), which isincorporated by reference herein.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

The present description relates generally to a system and method,generally referred to as a system, for determining the quality of querysuggestions using a network of users and advertisers, enabling theprioritization of high quality query suggestions to optimize theallocation of online advertisements.

BACKGROUND

Online advertising may be an important source of revenue for enterprisesengaged in electronic commerce. The advent of search engines may haveresulted in an increase in the use of sponsored search, or paid search,by advertisers. Sponsored search may be an arrangement where companiesand/or individuals pay a service provider for placement of theiradvertisement listing. The advertisement listing may be placed in asearch result set generated by the service provider's search engine ormay be placed on a page of a partner of the service provider, e.g., ablog. An advertiser may place bids for one or more keywords within asearch term bidding marketplace that may work in conjunction with one ormore search engines. An advertiser may bid on keywords that may indicatean interest in the products, services, information, etc. beingadvertised in the advertisement. The amount an advertiser may bid on thekeywords may indicate the cost the advertiser may be willing to pay forplacement of the advertisement.

A user may submit a query comprising one or more keywords to a searchengine and the search engine may produce a result set comprising one ormore listings that may fall within the scope of the query, includingsponsored search listings. The search engine may use the keywords, aswell as other features such as user and advertiser information, toselect sponsored search listings for inclusion in the result set. Theuser may generate a lead for an advertiser when the user selects thesponsored listing of the advertiser, such as by clicking on theadvertisement.

Search engines may strive to maintain an increasing supply of users todeliver valuable leads to advertisers and advertisers, in turn, maydemand a growing supply of leads from search engines. This may result ingrowth of search engine usage and online advertising budgets. Searchengines may retain and increase their supply of users by providingrelevant web search results and advertising. Advertisers may increasetheir demand of leads as lead quality and targeting improve. Amarketplace therefore may exist that includes a given keyword, the setof one or more users who may provide search queries comprising thekeyword over a given period of time (“lead supply”) and the advertiserswho may compete for leads (or clicks) for the given keyword. Searchengines or other advertisement providers may use the above-describedterm bidding marketplace, which is a form of an auction, to allocateleads to advertisers.

In a “dense” marketplace, advertiser demand may exceed the supply ofleads. The auction may be designed such that advertisers who are mostrelevant to the keyword, and/or value the lead the most, may place thehighest bid on the keyword. In “shallow” or “sparse” marketplaces,advertiser demand may not exceed the supply of leads. A shallowmarketplace may have limited leads because the marketplace may becharacterized by multiple keyword phrases, as well as keywords that maybe obscure and/or may have a very narrow context or intent. Since theremay be only a small number of advertisers bidding for these keywords,the average cost per click for a given lead may be generally low.Advertisers may bombard search engines with low bids for a large numberof such keywords to capture opportunities in shallow marketplaces. Theimbalances of supply and demand may lead to inadequate overall relevanceto users and a lack of competition among advertisers, ultimatelyresulting in a decrease in revenue to the service provider.

Furthermore the term-bidding marketplace may require advertisers topredict keywords or queries that may be searched for by users. If a usersearches for a keyword or query which has not been bid on by anyadvertisers, the search engine may not display any advertisements to theuser. If a search results page is displayed to a user with noadvertisements, there may be little likelihood of leads for theadvertisers and revenue for the search engine provider.

SUMMARY

A system for determining the quality of query suggestions using anetwork of users and advertisers may include a memory and a processor.The memory may be operatively connected to the processor and may store ahistorical dataset, a residual value, a query-advertisement link value,a query suggestion value, and a data representing a network. The networkmay comprise a plurality of query items representing queries linked to aplurality of advertisement items representing advertisements via aplurality of query-advertisement link items. Each query-advertisementlink item may comprise a weight representing the strength of theassociation between each linked query item and each linked advertisementitem, an advertisement value representing a value provided by the linkedadvertisement item, a suggested query data describing a query suggestedby a query suggestion system for the query item, and a match typerepresenting the query suggestion system. The processor may beoperatively connected to the memory and may retrieve the historicaldataset and generate the data representing the network. The processormay identify a query-advertisement link item comprising a suggestedquery data in the data representing the network. The processor maycalculate the residual value of the query suggestion system representedby the match type of the query-advertisement link item. The calculationof the residual value may be based on the weight and the advertisementvalue of each query-advertisement link sharing a match type and asuggested query data with the identified query-advertisement link item.The processor may calculate the query-advertisement link value for thequery-advertisement link. The calculation may be based on the weight andthe advertisement value of the query-advertisement link. The processormay determine a query suggestion value by adding the residual value tothe query-advertisement link value. The processor may store the querysuggestion value in the memory.

Other systems, methods, features and advantages will be, or will become,apparent to one with skill in the art upon examination of the followingfigures and detailed description. It is intended that all suchadditional systems, methods, features and advantages be included withinthis description, be within the scope of the embodiments, and beprotected by the following claims and be defined by the followingclaims. Further aspects and advantages are discussed below inconjunction with the description.

BRIEF DESCRIPTION OF THE DRAWINGS

The system and/or method may be better understood with reference to thefollowing drawings and description. Non-limiting and non-exhaustivedescriptions are described with reference to the following drawings. Thecomponents in the figures are not necessarily to scale, emphasis insteadbeing placed upon illustrating principles. In the figures, likereferenced numerals may refer to like parts throughout the differentfigures unless otherwise specified.

FIG. 1 is a block diagram of a system for generating query suggestionsusing a network of users and advertisers.

FIG. 2 is block diagram of a simplified view of a network environmentimplementing the system of FIG. 1 or other systems for generating querysuggestions using a network of users and advertisers.

FIG. 3 is a block diagram illustrating a system for generating querysuggestions using a network of users and advertisers.

FIG. 4 is a graph illustrating an example of a network of users andadvertisers used in the system of FIG. 3 or other systems for optimizingthe performance of online advertisements using a network of users andadvertisers.

FIG. 5 is a flowchart illustrating the operations of the system of FIG.3, or other systems for generating query suggestions using a network ofusers and advertisers.

FIG. 6 is a flowchart illustrating the operations of identifying the rawcontext data for a query/advertisement pairing in the system of FIG. 3,or other systems for generating query suggestions using a network ofusers and advertisers.

FIG. 7 is a flowchart illustrating the operations of building a linkbetween a query and an advertisement in the system of FIG. 3 or othersystems for generating query suggestions using a network of users andadvertisers.

FIG. 8 is a flowchart illustrating the use of a network of users andadvertisers, built by the system of FIG. 3 or other systems forgenerating query suggestions using a network of users and advertisers,to suggest queries related to a query.

FIG. 9 is a flowchart illustrating the use of a data structurerepresenting a network of users and advertisers, built by the system ofFIG. 3 or other systems for generating query suggestions using a networkof users and advertisers, to determine ad listings relevant to a query.

FIG. 10 is a flowchart illustrating the use of a data structurerepresenting a network of users and advertisers, built by the system ofFIG. 3 or other systems for generating query suggestions using a networkof users and advertisers, to determine the value of a suggested query.

FIG. 11 is a flowchart illustrating the use of a data structurerepresenting a network of users and advertisers, built by the system ofFIG. 3 or other systems for generating query suggestions using a networkof users and advertisers, to determine the value of a matching systemand a suggested query.

FIG. 12 is a flowchart illustrating the use of a data structurerepresenting a network of users and advertisers, built by the system ofFIG. 3 or other systems for generating query suggestions using a networkof users and advertisers, to integrate valuable query suggestions withexperimental query suggestions.

FIG. 13 is an illustration of an exemplary page displayingadvertisements.

FIG. 14 is a screenshot of a search results page displayingadvertisements.

FIG. 15 is an illustration a general computer system that may be used inthe system of FIG. 3 or other systems for generating query suggestionsusing a network of users and advertisers.

DETAILED DESCRIPTION

A system and method, generally referred to as a system, may relatedetermining the quality of query suggestions using a network of usersand advertisers, enabling the prioritization of high quality querysuggestions to optimize the allocation of online advertisements. Theprinciples described herein may be embodied in many different forms.

The system may build a data structure representing a network of usersand advertisers based on advertiser intent described by target queries,valuation and spend, as well as historical user behavior described byqueries, user profiles and other context. A query may refer to a set ofterms searched for by a user or a set of terms related to the content ofa page, such as a web page displayed to a user. The network may beindependent of the language and other regional characteristics of theunderlying data, enabling a plurality of networks to be combined acrossmarkets defined by language and other regional characteristics.

The network may be used to identify advertisements to be served by asearch engine, such as supplemental advertisements related to the user'ssearch query. The network may be used to estimate the relative qualityof advertisements and evolve a quality benchmark, such as a qualitybenchmark based on advertisement performance and/or user feedback. Theadditional advertisements may increase the depth and competitiveness ofshallow keywords by eliciting/inducing more overall user attention.

The network may further be used to generate keyword suggestions to bequeried at advertisement serving time, and/or to be presented toadvertisers during campaign management. The network may be used toevaluate the quality (relevance, value) of keywords suggested throughthe use of the network, and keywords suggested through other matchingtechniques, in the first and higher orders. The network may be used toutilize high quality keyword suggestions and further to explore unknownor low value suggestions scheduled by some measure based on a relevancemodel. The keyword suggestions may increase the depth andcompetitiveness of shallow keywords by eliciting/inducing more overalluser attention.

The network may be utilized in several ways to suggest keywords andidentify advertisements. The network may be used capture the semanticknowledge gap between raw user queries (often syntactically different)and underlying implicit user intent in an automated, non-intrusive,implicit way. The captured semantic knowledge gap may be utilized tosuggest keywords and/or identify advertisements.

The network may be analyzed to identify both significantly unrelated andsignificantly related sub-networks based on some affinity measure, suchas keyword semantic affinities, advertiser online spent, and pasthistorical performance based on user clicks and/or revenue generated.The sub-network relationships may be utilized to suggest keywords and/oridentify advertisements.

The network may be extendable to account for new emerging forms ofadvertisement performance feedback, such as clicks, various forms ofconversions, and/or any other metric for measuring advertisementperformance. Single or multiple forms of advertisement feedback may betranslated into semantic knowledge. The semantic knowledge may beutilized to suggest keywords and/or identify advertisements.

The network may be adaptable to account for temporal increments givenincreasing advertiser participation, changes in advertiser intent,valuation and online spend, changes in user behavior, demographics andmix, changes in aggregate user intent, search usage, and mix. Theadapted network may be capable of capturing temporal shifts in userintent, advertiser intent and/or other context and a corresponding shiftin the underlying semantic knowledge. The shift in semantic knowledgemay be utilized to suggest keywords and/or identify advertisements, suchas by implicitly capturing language seasonal patterns, and languageusage patterns.

FIG. 1 provides a general overview of a system 100 for generating querysuggestions using a network of users and advertisers. Not all of thedepicted components may be required, however, and some implementationsmay include additional components. Variations in the arrangement andtype of the components may be made without departing from the spirit orscope of the claims as set forth herein. Additional, different or fewercomponents may be provided.

The system 100 may include one or more revenue generators 110A-N, suchas advertisers, a service provider 130, such as a search enginemarketing service provider, and one or more users 120A-N, such as websurfers or consumers. The service provider 130 may implement anadvertising campaign management system incorporating an auction basedand/or non-auction based advertisement serving system. The revenuegenerators 110A-N may pay the service provider 130 to serve, or display,advertisements of their goods or services, such as on-lineadvertisements, on a network, such as the Internet. The advertisementsmay include sponsored listings, banners ads, popup advertisements, orgenerally any way of attracting the users 120A-N to the web site of therevenue generators 110A-N.

The amount the revenue generators 110A-N may pay the service provider130 may be based on one or more factors. These factors may includeimpressions, click throughs, conversions, and/or generally any metricrelating to the advertisement and/or the behavior of the users 120A-N.The impressions may refer to the number of times an advertisement mayhave been displayed to the users 120A-N. The click throughs may refer tothe number of times the users 120A-N may have clicked through anadvertisement to a web site of one of the revenue generators 110A-N,such as the revenue generator A 110A. The conversions may refer to thenumber of times a desired action was taken by the users 120A-N afterclicking though to a web site of the revenue generator A 110A. Thedesired actions may include submitting a sales lead, making a purchase,viewing a key page of the site, downloading a whitepaper, and/or anyother measurable action. If the desired action is making a purchase,then the revenue generator A 110A may pay the service provider 130 apercentage of the purchase.

The users 120A-N may be consumers of goods or services who may besearching for a business, such as the business of one of the revenuegenerators 110A-N. Alternatively or in addition the users 120A-N may bemachines or other servers, such as the third party server 250. The users120A-N may supply information describing themselves to the serviceprovider 130, such as the location, gender, or age of the users 120A-N,or generally any information that may be required for the users 120A-Nto utilize the services provided by the service provider 130.

In the system 100, the revenue generators 110A-N may interact with theservice provider 130, such as via a web application. The revenuegenerators 110A-N may send information, such as billing, website andadvertisement information, to the service provider 130 via the webapplication. The web application may include a web browser or otherapplication, such as any application capable of displaying web content.The application may be implemented with a processor such as a personalcomputer, personal digital assistant, mobile phone, or any other machinecapable of implementing a web application.

The users 120A-N may also interact individually with the serviceprovider 130, such as via a web application. The users 120A-N mayinteract with the service provider 130 via a web based application or astandalone application. The service provider 130 may communicate data tothe revenue generators 110A-N and the users 120A-N over a network. Thefollowing examples may refer to a revenue generator A 110A as an onlineadvertiser; however the system 100 may apply to any revenue generators110A-N who may benefit from a network of users and advertisers, such asa service provider partner.

One example of a service provider partner may be a content publisher.Content publishers may be service provider partners who may displaycontent, such as news articles, videos, or any other type of content tothe users 120A-N. Along with the content, content publishers may displayadvertisements of the advertisers to the users 120A-N. The serviceprovider 130 may supply the advertisements to the content publishers.The advertisements may relate to the content displayed on the page, orthe advertisements may relate to the characteristics, demographicsand/or login-profiles of the users 120A-N. When the users 120A-Ninteract with an advertisement of one of the advertisers, theadvertisers may pay the service provider 130. The service provider 130may in turn pay the content publisher. Thus the revenue generators110A-N may include one or more content publishers, advertisers, and/orother service provider partners.

In operation, one of the revenue generators 110A-N, such as revenuegenerator A 110A, may provide information to the service provider 130.This information may relate to the transaction taking place between therevenue generator A 110A and the service provider 130, or may relate toan account the revenue A 110A generator maintains with the serviceprovider 130. In the case of a revenue generator A 110A who is an onlineadvertiser, the revenue generator A 110A may provide initial informationnecessary to open an account with the service provider 130. The revenuegenerators 110A-N may implement one or more advertising tactics with theservice provider 130 to target advertisements to the users 120A-N and/orthe revenue generators 110A-N may authorize the service provider 130 touse any advertising tactic, or method, to display their advertisementsto the users 120A-N.

One example of an advertising tactic may be sponsored search, such astargeting advertisements to search terms or keywords. Sponsored searchmay operate within the context of an auction-based system or marketplacethat may be used by the revenue generators 110A-N to bid for searchterms or queries. When the terms are used in a search, the ad listingsor links of a revenue generator, such as the revenue generator A 110A,may be displayed among the search results. Revenue generators 110A-N mayfurther bid for position or prominence of their listings in the searchresults. With regard to auction-based sponsored search, the revenuegenerator A 110A may provide a uniform resource locator (URL) for thewebpage to which the ad may take the users 120A-N to if clicked on. Therevenue generator A 110A may also provide the text or creative of theadvertisement that may be displayed in connection with the URL. Arevenue generator A 110A may identify one or more terms that may beassociated with the advertisement.

Another example of an advertising tactic may be content matching.Content match advertisements may be used by the revenue generator A 110Ato complement, or as alternative to, the sponsored search tactic. Adsstored according to the content match tactic may be displayed alongsiderelevant articles, product reviews, etc, presented to the users 120A-Nby the service provider 130 or a service provider partner, such as acontent publisher. The system 100 may implement a content matchingsystem. The content matching system may process the words on a givenpage to determine a set of terms. The set of terms may be the mostcommonly occurring words, or may be determined by some other factor. Theset of terms may then be used to determine which of the content matchadvertisements to display. The content matching system may use the setof terms to select advertisements, such as by selecting theadvertisements which contain the most number of words matching the setof terms. The set of terms may be referred to as a query or a contentmatch query.

Content match advertisements may be displayed on any web page containingcontent relevant to the advertisement. For the content match tactic, therevenue generator A 110A may provide one or more URLs identifying theaddress of a webpage a given ad may take the users 120A-N to if clickedon. The revenue generator A 110A may also provide the text, image, videoor other type of multimedia comprising the creative portion of theadvertisement that may be displayed next to the URL.

Another example of an advertising tactic may be a banner advertisementor popup advertisement. The banner ad and/or popup ad tactic may be usedby the revenue generators 110A-N to complement, or as alternative to,the sponsored search tactic and the content match tactic. In contrast tothe sponsored search tactic and content match tactic, which may be basedon a pay-per-click payment scheme, a revenue generator 110A-N may payfor every display of a banner ad and/or popup ad, referred to as animpression. Alternatively, if the banner ad and/or popup ad displays aphone number, a revenue generator, such as the revenue generator A 110Amay only be billed if a user, such as the user A 120A, calls the phonenumber associated with the advertisement (“pay-per-call”). Thus, for thebanner ad and/or popup ad tactic, the revenue generator A 110A mayprovider a URL to the webpage where the ad may take the user A 120A ifclicked on, as well as the creative or the given banner ad and/or popupad.

A revenue generator A 110A who is an online advertiser may maintainseveral accounts with the service provider 130. For each account therevenue generator A 110A may maintain several advertising campaigns,such as an MP3 player campaign, a car campaign, or any otherdistinguishable category of products and/or services. Each campaign mayinclude one or more ad groups. The ad groups may further distinguish thecategory of products and/or services represented in the advertisingcampaign, such as by search tactic, performance parameter, demographicof user, family of products, or almost any other parameter desired bythe revenue generators 110A-N.

For example, if the advertising campaign is for MP3 Players, there maybe an ad group each brand of MP3 players, such as APPLE IPOD® orMICROSOFT ZUNE®. Allowing the revenue generators 110A-N to determinetheir own ad groups may allow the service provider 130 to provide moreuseful information to the revenue generators 110A-N. The revenuegenerators 110A-N may thereby display, manage, optimize, or view reportson, advertisement campaign information in a manner most relevant to arevenue generator, such as the revenue generator A 110A.

The ad groups may include one or more listings. A listing may include atitle, a description, one or more search keywords, an advertisement, adestination URL, and a bid amount. A listing may represent anassociation between the one or more search keywords identified by therevenue generator A 110A, and an advertisement of the revenue generatorA 110A.

The title may be the name of the product being advertised, such as “JEEPWRANGLER®.” The description may describe the product being advertised.For example, if DAIMLERCHRYSLER® wished to advertise a DAIMLERCHRYSLERJEEP WRANGLER®, the listing may have a description of “DAIMLERCHRYSLERJEEP WRANGLER®,” “JEEP WRANGLER®,” or “5 PASSENGER JEEP WRANGLER®.”

The destination URL may represent the link the revenue generator A 110Awishes a user A 120A to be directed to upon clicking on theadvertisement of the revenue generator A 110A, such as the home page ofthe revenue generator A 110A. The bid amount may represent a maximumamount the revenue generator A 110A may be willing to pay each time auser A 120A may click on the advertisement of the revenue generator A110A or each time the advertisement of the revenue generator A 110A maybe shown to a user A 120A.

The keywords may represent one or more search terms that the revenuegenerator A 110A may wish to associate their advertisement with. When auser A 120A searches for one of the listing's keywords, theadvertisement of the revenue generator A 110A may be displayed on thesearch results page.

For example, a revenue generator A 110A, such as DAIMLERCHRYSLER®, maydesire to target an online advertisement for a CHRYSLER JEEP WRANGLER®to users 120A-N searching for the keywords “JEEP®”, “WRANGLER®”, or“JEEP WRANGLER®”. DAIMLERCHRYSLER® may place a bid with the serviceprovider 130 for the search keywords “JEEP®”, “WRANGLER®”, and “JEEPWRANGLER®” and may associate the online advertisement for aDAIMLERCHRYSLER JEEP WRANGLER® with the keywords. The advertisement ofthe revenue generator A 110A may be displayed when one of the users120A-N searches for the keywords “JEEP®”, “WRANGLER®”, or “JEEPWRANGLER®”.

Alternatively or in addition, the service provider 130 may implement aquery suggestion system. A query suggestion system may perform ananalysis on the query of the user A 120A, or the query determined from,or related to, the content of page, such as a web page displayed to theuser A 120A, to find additional queries that may relate to the query ofthe user A 120A, or the query determined from the content of a page. Ifadditional queries are found, advertisements with bids on any of theadditional queries may be displayed to the user A 120A in addition tothe advertisements with bids on the original query. Thus the user A 120Amay click on an advertisement of a revenue generator A 110A who did notbid on the query the user A 120A searched for, or the query determinedfrom the content of a page, but a query matched, by a query suggestionsystem, to the query searched for by the user A 120A. Some examples ofquery suggestion systems may include King Kong, SPM, MOD, Units, orquery suggestions derived from a network of users and advertisers.

More detail regarding the aspects of query suggestions systems, as wellas their structure, function and operation, can be found in commonlyowned U.S. patent application Ser. No. 10/625,082, filed on Jul. 22,2003, entitled, “TERM-BASED CONCEPT MARKET”; U.S. patent applicationSer. No. 11/295,166, filed on Dec. 5, 2005, entitled “SYSTEMS ANDMETHODS FOR MANAGING AND USING MULTIPLE CONCEPT NETWORKS FOR ASSISTEDSEARCH PROCESSING”; U.S. patent application Ser. No. 10/797,586, filedon Mar. 9, 2004, entitled “VECTOR ANALYSIS OF HISTOGRAMS FOR UNITS OF ACONCEPT NETWORK IN SEARCH QUERY PROCESSING”; U.S. patent applicationSer. No. 10/797,614, filed on Mar. 9, 2004, entitled “SYSTEMS ANDMETHODS FOR SEARCH PROCESSING USING SUPERUNITS”; U.S. Pat. No.7,051,023, filed on Nov. 12, 2003, entitled “SYSTEMS AND METHODS FORGENERATING CONCEPT UNITS FROM SEARCH QUERIES,” and U.S. Pat. No.6,876,997, filed on May 22, 2000, entitled “METHOD AND APPARATUS FORIDENTIFYING RELATED SEARCHES IN A DATABASE SEARCH SYSTEM, all of whichare hereby incorporated herein by reference in their entirety. Thesystems and methods herein associated with query suggestion systemsanalysis may be practiced in combination with methods and systemsdescribed in the above-identified patent applications incorporated byreference.

An advertisement may represent the data the revenue generator A 110Awishes to be displayed to a user A 120A when the user A 120A searchesfor one of the listing's keywords. An advertisement may include acombination of the description and the title. The ad groups may eachcontain several different advertisements, which may be referred to ascreatives. Each of the individual advertisements in an ad group may beassociated with the same keywords. The advertisements may differslightly in creative aspects or may be targeted to differentdemographics of the users 120A-N.

There may be some instances where multiple revenue generators 110A-N mayhave bid on the same search keyword. The service provider 130 may serveto the users 120A-N the online advertisements that the users 120A-N maybe most likely to click on. For example, the service provider 130 mayinclude a relevancy assessment to determine the relevancy of themultiple online advertisements to the search keyword. The more relevantan advertisement may be to the keyword the more likely it may be thatthe user A 120A may click on the advertisement. The relevancy may bedetermined by the service provider 130 or a third party relevancyengine.

When one of the users 120A-N, such as the user A 120A, interacts withthe service provider 130, such as by searching for a keyword, theservice provider 130 may retain data describing the interaction with theuser A 120A. The stored data may include the keyword searched for, thegeographic location of the user A 120A, and the date/time the user A120A interacted with the service provider 130. Further the data mayinclude data describing the number of prominent ads, or top adsdisplayed on the page to the user A 120A. FIGS. 13 and 14 may showexamples of top ads. The number of top ads on a given page may bereferred to as the “DUDE” state of the page. The service provider 130may retain the DUDE state of a page or query when a user A 120A clickson an advertisement. The stored data may also generally include any dataavailable to the service provider 130 that may assist in describing theinteraction with the user A 120A, or describing the user A 120A.

The service provider 130 may also store data that indicates whether anadvertisement of one of the revenue generators 110A-N, such as therevenue generator A 110A was displayed to the user A 120A, and whetherthe user A 120A clicked on the advertisement, or generally any otherdata that may assist the revenue generators 110A-N in determining theeffectiveness of their advertisements. The data may also include datadescribing the rank of the advertisement clicked on by the user A 120A.The rank may refer to the order in which the advertisements aredisplayed on the page. For example, the first displayed advertisementmay have a rank of “1,” the second displayed advertisement may have arank of “2,” and so on.

In some instances the advertisement may have been displayed to the userA 120A as a result of a query suggestion from a query suggestion system,or matching system, implemented by the service provider 130. The querysuggestion system may have suggested a query matching the query of theuser A 120A. The suggested query may have had advertisements relevant tothe query of the user A 120A and the relevant advertisement may havebeen displayed to the user A 120A. In theses instances, the serviceprovider 130 may store the query that the service provider 130 matchedto the query of the user A 120A along with a unique identifierdescribing the matching system that suggested the query, such as thename of the matching system.

The users 120A-N may supply information relating to their geographiclocation and/or other descriptive information upon their initialinteraction with the service provider 130. Alternatively or in additionthe service provider 130 may obtain the location of the user A 120Abased on the IP address of the user A 120A. The service provider 130 mayuse a current date/time stamp to store the date/time when the user A120A interacted with the service provider 130.

The service provider 130 may generate reports based on the datacollected from the user interactions and communicate the reports to therevenue generators 110A-N to assist the revenue generators 110A-N inmeasuring the effectiveness of their online advertising. The reports mayindicate the number of times the users 120A-N searched for the keywordsbid on by the revenue generators 110A-N, the number of times eachadvertisement of the ad groups of the revenue generators 110A-N wasdisplayed to the users 120A-N, the number of times the users 120A-Nclicked through on each advertisement of the ad groups of the revenuegenerators 110A-N, and/or the number of times a desired action wasperformed by the users 120A-N after clicking through on anadvertisement. The reports may also generally indicate any data that mayassist the revenue generators 110A-N in measuring or managing theeffectiveness of their online advertising.

The reports may further include sub-reports that segment the data intomore specific categories, including the time intervals when theinteractions occurred, such as weeknights primetime, weekends, etc., thedemographics of the users 120A-N, such as men ages 18-34, the locationof the users 120A-N. The reports may also generally include any otherdata categorization that may assist the revenue generators 110A-N indetermining the effectiveness of their online advertising.

More detail regarding the aspects of auction-based systems, as well asthe structure, function and operation of the service provider 130, asmentioned above, can be found in commonly owned U.S. patent applicationSer. No. 10/625,082, filed on Jul. 22, 2003, entitled, “TERM-BASEDCONCEPT MARKET”; U.S. patent application Ser. No. 10/625,000, file onJul. 22, 2003, entitled, “CONCEPT VALUATION IN A TERM-BASED CONCEPTMARKET” filed on Jul. 22, 2003; U.S. patent application Ser. No.10/625,001, filed on Jul. 22, 2003, entitled, “TERM-BASED CONCEPTINSTRUMENTS”; and U.S. patent application Ser. No. 11/489,386, filed onJul. 18, 2006, entitled, “ARCHITECTURE FOR AN ADVERTISEMENT DELIVERYSYSTEM,” all of which are hereby incorporated herein by reference intheir entirety. The systems and methods herein associated with adcampaign management may be practiced in combination with methods andsystems described in the above-identified patent applicationsincorporated by reference.

FIG. 2 provides a simplified view of a network environment 200implementing the system of FIG. 1 or other systems for generating querysuggestions using a network of users and advertisers. Not all of thedepicted components may be required, however, and some implementationsmay include additional components not shown in the figure. Variations inthe arrangement and type of the components may be made without departingfrom the spirit or scope of the claims as set forth herein. Additional,different or fewer components may be provided.

The network environment 200 may include one or more web applications,standalone applications and mobile applications 210A-N, which may becollectively or individually referred to as client applications for therevenue generators 110A-N. The system 200 may also include one or moreweb applications, standalone applications, mobile applications 220A-N,which may collectively be referred to as client applications for theusers 120A-N, or individually as a user client application. The system200 may also include a network 230, a network 235, the service providerserver 240, a data store 245, a third party server 250, and anadvertising services server 260.

Some or all of the advertisement services server 260, service providerserver 240, and third-party server 250 may be in communication with eachother by way of network 235. The advertisement services server 260,third-party server 250 and service provider server 240 may eachrepresent multiple linked computing devices. Multiple distinct thirdparty servers, such as the third-party server 250, may be included inthe network environment 200. A portion or all of the advertisementservices server 260 and/or the third-party server 250 may be a part ofthe service provider server 240.

The data store 245 may be operative to store data, such as data relatingto interactions with the users 120A-N. The data store 245 may includeone or more relational databases or other data stores that may bemanaged using various known database management techniques, such as, forexample, SQL and object-based techniques. Alternatively or in additionthe data store 245 may be implemented using one or more of the magnetic,optical, solid state or tape drives. The data store 245 may be incommunication with the service provider server 240. Alternatively or inaddition the data store 245 may be in communication with the serviceprovider server 240 through the network 235.

The networks 230, 235 may include wide area networks (WAN), such as theinternet, local area networks (LAN), campus area networks, metropolitanarea networks, or any other networks that may allow for datacommunication. The network 230 may include the Internet and may includeall or part of network 235; network 235 may include all or part ofnetwork 230. The networks 230, 235 may be divided into sub-networks. Thesub-networks may allow access to all of the other components connectedto the networks 230, 235 in the system 200, or the sub-networks mayrestrict access between the components connected to the networks 230,235. The network 235 may be regarded as a public or private networkconnection and may include, for example, a virtual private network or anencryption or other security mechanism employed over the publicInternet, or the like.

The revenue generators 110A-N may use a web application 210A, standaloneapplication 210B, or a mobile application 210N, or any combinationthereof, to communicate to the service provider server 240, such as viathe networks 230, 235. Similarly, the users 120A-N may use a webapplication 220A, a standalone application 220B, or a mobile application220N to communicate to the service provider server 240, via the networks230, 235.

The service provider server 240 may communicate to the revenuegenerators 110A-N via the networks 230, 235, through the webapplications, standalone applications or mobile applications 210A-N. Theservice provider server 240 may also communicate to the users 120A-N viathe networks 230, 235, through the web applications, standaloneapplications or mobile applications 220A-N.

The web applications, standalone applications and mobile applications210A-N, 220A-N may be connected to the network 230 in any configurationthat supports data transfer. This may include a data connection to thenetwork 230 that may be wired or wireless. Any of the web applications,standalone applications and mobile applications 210A-N, 220A-N mayindividually be referred to as a client application. The webapplications 210A, 220A may run on any platform that supports webcontent, such as a web browser or a computer, a mobile phone, personaldigital assistant (PDA), pager, network-enabled television, digitalvideo recorder, such as TIVO®, automobile and/or any appliance capableof data communications.

The standalone applications 210B, 220B may run on a machine that mayhave a processor, memory, a display, a user interface and acommunication interface. The processor may be operatively connected tothe memory, display and the interfaces and may perform tasks at therequest of the standalone applications 210B, 220B or the underlyingoperating system. The memory may be capable of storing data. The displaymay be operatively connected to the memory and the processor and may becapable of displaying information to the revenue generator B 110B or theuser B 120B. The user interface may be operatively connected to thememory, the processor, and the display and may be capable of interactingwith a user B 120B or a revenue generator B 110B. The communicationinterface may be operatively connected to the memory, and the processor,and may be capable of communicating through the networks 230, 235 withthe service provider server 240, third party server 250 and advertisingservices server 260. The standalone applications 210B, 220B may beprogrammed in any programming language that supports communicationprotocols. These languages may include: SUN JAVA®, C++, C#, ASP, SUNJAVASCRIPT®, asynchronous SUN JAVASCRIPT®, or ADOBE FLASH ACTIONSCRIPT®,amongst others.

The mobile applications 210N, 220N may run on any mobile device that mayhave a data connection. The data connection may be a cellularconnection, a wireless data connection, an internet connection, aninfra-red connection, a Bluetooth connection, or any other connectioncapable of transmitting data.

The service provider server 240 may include one or more of thefollowing: an application server, a data store, such as the data store245, a database server, a middleware server, and an advertising servicesserver. The service provider server 240 may co-exist on one machine ormay be running in a distributed configuration on one or more machines.The service provider server 240 may collectively be referred to as theserver. The service provider may implement a search engine marketingsystem and/or an advertising campaign management system. The serviceprovider server 240 may receive requests from the users 120A-N and therevenue generators 110A-N and may serve pages to the users 120A-N andthe revenue generators 110A-N based on their requests.

The third party server 250 may include one or more of the following: anapplication server, a data source, such as a database server, amiddleware server, and an advertising services server. The third partyserver may implement a relevancy engine, a context matching engine, orany other third party application that may be used in a search enginemarketing system and/or an advertising campaign management system. Thethird party server 250 may co-exist on one machine or may be running ina distributed configuration on one or more machines. The third partyserver 250 may receive requests from the users 120A-N and the revenuegenerators 110A-N and may serve pages to the users 120A-N and therevenue generators 110A-N based on their requests.

The service provider server 240, the third party server 250 and theadvertising services server 260 may be one or more computing devices ofvarious kinds, such as the computing device in FIG. 15. Such computingdevices may generally include any device that may be configured toperform computation and that may be capable of sending and receivingdata communications by way of one or more wired and/or wirelesscommunication interfaces. Such devices may be configured to communicatein accordance with any of a variety of network protocols, including butnot limited to protocols within the Transmission ControlProtocol/Internet Protocol (TCP/IP) protocol suite. For example, the webapplications 210A, 210A may employ HTTP to request information, such asa web page, from a web server, which may be a process executing on theservice provider server 240 or the third-party server 250.

There may be several configurations of database servers, such as thedata store 245, application servers, middleware servers and advertisingservices servers included in the service provider server 240, or thethird party server 250. Database servers may include MICROSOFT SQLSERVER®, ORACLE®, IBM DB2® or any other database software, relational orotherwise. The application server may be APACHE TOMCAT®, MICROSOFT IIS®,ADOBE COLDFUSION®, YAPACHE® or any other application server thatsupports communication protocols. The middleware server may be anymiddleware that connects software components or applications. Themiddleware server may be a relevancy engine, a context matching engine,or any other middleware that may be used in a search engine marketingsystem and/or an advertising campaign management system.

The application server on the service provider server 240 or the thirdparty server 250 may serve pages, such as web pages to the users 120A-Nand the revenue generators 110A-N. The advertising services server 260may provide a platform for the inclusion of advertisements in pages,such as web pages. The advertising services server 260 may also existindependent of the service provider server 240 and the third partyserver 250. The advertisement services server 260 may be used forproviding advertisements that may be displayed to users 120A-N on pages,such as web pages. The advertising services server 260 may implement asearch engine marketing system and/or an advertising campaign managementsystem.

The networks 230, 235 may be configured to couple one computing deviceto another computing device to enable communication of data between thedevices. The networks 230, 235 may generally be enabled to employ anyform of machine-readable media for communicating information from onedevice to another. Each of networks 230, 235 may include one or more ofa wireless network, a wired network, a local area network (LAN), a widearea network (WAN), a direct connection such as through a UniversalSerial Bus (USB) port, and the like, and may include the set ofinterconnected networks that make up the Internet. The networks 230, 235may include any communication method by which information may travelbetween computing devices.

FIG. 3 illustrates a system 300 for generating query suggestions using anetwork of users and advertisers. The system 300 may include an adserving system 310, a graph component 320, a service provider server240, a data store 245, and a network 235. The ad serving system 310 maybe implemented by the service provider server 240, the ad servicesserver 260, or the third party server 250. The ad serving system 310 maybe an auction-based ad serving system. The ad serving system 310 mayinclude an ad data store 318, a sponsored search server 312, a contentmatch server 316, and a redirect server 314. The graph component 320 maybe implemented by the service provider server 240, the ad servicesserver 260, or the third party server 250. The graph component 320 mayinclude a graph processor 322, a graph analyzer 324, and a graph datastore 326. The graph component 320 may exist on one machine or may berunning in a distributed configuration on one or more machines. The oneor more machines of the graph component 320 may be one or more computingdevices of various kinds, such as the computing device in FIG. 15. Notall of the depicted components may be required, however, and someimplementations may include additional components not shown in thefigure. Variations in the arrangement and type of the components may bemade without departing from the spirit or scope of the claims as setforth herein. Additional, different or fewer components may be provided.

The ad data store 318 may be operative to store data, such asadvertisement listings. The ad data store 318 may include one or morerelational databases or other data stores that may be managed usingvarious known database management techniques, such as, for example, SQLand object-based techniques. Alternatively or in addition the ad datastore 318 may be implemented using one or more of the magnetic, optical,solid state or tape drives.

The sponsored search server 312 may be operative to process sponsoredsearch listing requests from the client applications 210A-N, receivedvia the service provider server 240 or the graph component 320. When arequest for a sponsored search listing comes from service providerserver 240 or the graph component 320, the sponsored search server 312may query the ad data store 318 for any advertisements, matching thesearch terms specified in the request. If matching ad listings areavailable in the ad data store 318, the sponsored search server 312 mayreturn the retrieved data to the service provider server 240. Theservice provider server 240 may then serve the ad listings, such assponsored listings, to the client applications 210A-N. Theadvertisements may be displayed in descending order based on the bidvalue for the given search terms whereby matching ads with the highestbids are displayed first followed by the lower bid advertisements.Alternatively or in addition the advertisements may be displayed in theorder based on the relevancy of the advertisements to the search terms.The relevancy may be determined by a relevancy engine implemented on theservice provider server 240 or the third party server 250.

The content match server 316 may operate in a similar manner. Thecontent match server 316 may be operative to process content matchlisting requests from the service provider server 240 or the graphcomponent 320. When a request for a content match listing comes from theservice provider server 240 or the graph component 320, the contentmatch server 316 may query the ad data store 318 for any advertisementsmatching the search terms specified in the request. If matching adlistings are available in the ad data store 318, the content matchserver 316 may return the data the service provider server 240. Theservice provider server 240 may then serve the advertisements to theclient applications 210A-N. The advertisements may be displayed indescending order based on the bid value for the given search termswhereby matching ads with the highest bids are displayed first followedby the lower bid advertisements. Alternatively or in addition theadvertisements may be displayed in the order based on the relevancy ofthe advertisements to the search terms. The relevancy may be determinedby a relevancy engine implemented on the service provider server 240 orthe third party server 250.

The graph component 320 may be operative to build, store, and analyzedata representing a graph through the graph processor 322, the graphanalyzer 324, and the graph data store 326. The graph may be a datarepresentation of a network of users and advertisers throughrelationships between advertisements and queries. A query may refer tothe set of terms searched for by one of the users 120A-N, or a set ofterms that may be related to the content on a page displayed to one ofthe users 120A-N.

The graph data store 326 may be operative to store data, such as datadescribing a network of users and advertisers, or advertisements andqueries. The graph data store 326 may include one or more relationaldatabases or other data stores that may be managed using various knowndatabase management techniques, such as, for example, SQL andobject-based techniques. Alternatively or in addition the graph datastore 326 may be implemented using one or more of the magnetic, optical,solid state or tape drives.

The graph processor 322 may be operative to process historical data,such as historical click data to generate data describing a network ofusers and advertisers, as illustrated below in FIG. 4. The network ofusers and advertisers may be represented by data describingrelationships between advertisements and queries.

The graph processor 322 may store the graph data in the graph data store326. The graph processor 322 may retrieve the historical data from thedata store 245 to generate the graph data. The graph processor 322 maybe in communication with the data store 245, or may access the datastore 245 via the service provider server 240.

The graph processor 322 may build the graph by processing the historicaldata. The historical data may be processed to build link data describingthe relationships between the queries, such as search queries of theusers 120A-N and/or queries, or a set of terms, related to the contenton a page displayed to the user 120A-N, and the advertisements displayedas a result of the queries. The links may be weighted by a metricdescribing the effectiveness of the advertisement, such as data relatedto user click throughs, conversions, or any other metric measuring theeffectiveness of the online advertisements. The graph may be independentof the language and other regional characteristics of the underlyingdata. The graph processor 322 may be capable of generating the graph byusing any of the aforementioned metrics measuring the effectiveness ofonline advertisements.

Alternatively or in addition the graph processor 322 may only generate alink between a query and an advertisement if one of the users 120A-Nclicked through on the advertisement. Therefore if an advertisement wasdisplayed to the users 120A-N for a particular query and none of theusers 120A-N clicked on the advertisement during the period of timerepresented by the historical data then the graph processor 322 may notgenerate a link for the advertisement/query pair.

The graph processor 322 may re-process the historical data to build anew graph at set intervals, such as daily, weekly, monthly, or any otherperiod that may increase the accuracy of the graph's representation ofthe network. The graph data store 326 may store every build of thegraph, identifying each individual build by the date/time the buildoccurred.

The graph analyzer 324 may be operative to analyze the stored graph datato perform a specified task, such as supplying suggested search termsrelated to the terms searched for by one of the users 120A-N, supplyingadvertisements related to the terms search for, or any other task thatmay be accomplished by analyzing the graph data. The graph analyzer 324may analyze the graph in real time, such as when a search term isreceived from the service provider.

Alternatively or in addition, the graph analyzer 324 may pre-process thegraph data to generate a separate data structure. The data structure maybe hashmap linking each query to relevant queries and advertisements.Large scale implementations of the network may require offlinepre-processing of the graph data.

The graph analyzer 324 may be operative to analyze the graph to increasethe depth and competitiveness of keywords using the graph. The graphanalyzer 324 may be operative to analyze the graph to generate keywordsuggestions which may be queried at advertisement serving time,presented to advertisers during campaign management and/or added toaugment advertisements to be served by the service provider server 240.

The graph analyzer 324 may be operative to analyze the graph to evaluatethe quality (relevance, value) of keyword suggestions and other matchingtechniques in the first and higher orders. The graph analyzer 324 may beoperative to analyze the graph to determine high performing suggestionsand to explore unknown or low value suggestions scheduled by somemeasure based on relevance. The graph analyzer 324 may be operative toanalyze the graph to estimate the relative quality of advertisements.

The graph analyzer 324 may be operative to analyze the graph to capturethe semantic knowledge gap between raw user queries (often syntacticallydifferent) and underlying user intent behind the queries. The graphanalyzer 324 may use the semantic knowledge gap to generate keywordsuggestions. For example, each query may describe an intent and/or needof a user A 120A in the form of a set of keywords. The user intentand/or need may not be accurately represented by the queries since thequeries may only partially capture the semantic intent of the user A120A.

The graph analyzer 324 may associate the queries of the user A 120A withrelevant queries of the other users 120B-N per the calculations below.The graph analyzer 324 may organize the queries into groups. Themembership of a query q in a group may be determined by the number ofrelevant queries that the query q shares with other queries that may bea member of the group. Membership to a group may be partial. For examplethe query q may belong to group X at 70% and to group Y at 30%.Membership to a group may be determined by any data clusteringalgorithm, such as k-means clustering, QT clustering, or Fuzzy c-meansclustering.

Once the allocation of queries to each group has been completed thesalient queries in each group may be used to describe the semanticintent of the user A 120A. The salient query of each group may also bedetermined by utilizing data clustering algorithms, such as k-meansclustering, QT clustering, or Fuzzy c-means clustering. For example, auser query such as “mp3 player” may have relevant queries of “ipod” ®and “noise-canceling headphones.” The query “ipod” ® may belong to thequery group described by “portable music players” while the query“noise-canceling headphones may belong to the query group described by“music players accessories.”

The graph analyzer 324 may then determine the relationship value betweenthe user query and the salient queries of each group. The groups thatare found to be closely related to the user query may capture thesemantic intent of the user A 120. The user query may then be matchedwith queries and advertisements associated with these groups.

Alternatively or in addition to the groups may be organized by bothqueries and advertisements that the users 120A-N clicked on. The salientrepresentative may be either queries or advertisements.

The graph analyzer 324 may be operative to analyze each successive buildof the graph to determine advertiser and/or user changes, such aschanges in advertiser participation, advertiser intent, advertiservaluation and spend, user behavior, demographics and mix, aggregate userintent and mix, and search usage. The graph analyzer 324 may beoperative to combine various builds of the graph across markets definedby language and other regional characteristics.

The graph analyzer 324 may be operative to analyze successive builds ofthe graph to capture temporal shifts in user intent, advertiser intentand/or context. The graph analyzer 324 may use the captured temporalshift to identify a corresponding shift in the semantic knowledge in theform of keyword suggestions. The keyword suggestions may implicitlycapture language seasonal patterns, and progress of human knowledgerepresentation in the form of language.

The graph analyzer 324 may be operative to identify both significantlyrelated and unrelated sub-networks within the network represented by thegraph. The sub-networks may be identified based on keyword semanticaffinities, advertiser online spend, and/or historical performance basedon user interactions and revenue generated by the advertisements. Inaddition, the graph analyzer 324 may be operative to clustering groupsof related nodes. For example, the relationship or proximity of queriesand advertisements may be determined, as demonstrated below. The queriesthat are determined to be closely related may be grouped into a node.The advertisements related to those queries may also be grouped into anode. These nodes may provide a higher level perspective of the network.Alternatively or in addition advertisements of a common advertiser maybe grouped together. This network may be used to determine information,such as demographics, about the users 120A-N interested in a certainadvertiser, regardless of the specific advertisement.

FIG. 4 is a graph 400 illustrating an example of a network of users andadvertisers used in by the system 300 of FIG. 3 or other systems forgenerating query suggestions using a network of users and advertisers.The graph 400 may be a bipartite graph. A bipartite graph may be a graphcontaining two types of nodes or points. In a bipartite graph no nodemay be linked to another node of the same type.

In the case of the graph 400, the node types may be query nodes andadvertisements nodes. The query nodes may represent queries performed bythe users 120A-N as represented in the historical data, queries relatedto content on a page displayed to the users 120A-N, and/or any other setof terms that may be matched to an advertisement. The query nodes mayrepresent the interest of the users 120A-N as demonstrated through thesearch queries. The advertisement nodes may represent the advertisementsthat may have been displayed to, or clicked on, by the users 120A-N as aresult of the queries. The advertisement nodes may represent the revenuegenerators 110A-N, such as advertisers, or more particularly theadvertisement nodes may represent the intent of the advertisers. Theadvertisers' intent may be demonstrated through the queries theadvertisements may be linked to and therefore the queries theadvertisers' may have previously bid on.

A query node may be linked to an advertisement node if one of the users120A-N, such as the user A 120A searches for the query and theadvertisement is displayed to the user A 120A as a result of the query.Alternatively or in addition a query node may be linked to anadvertisement node if an advertisement is displayed to the user A 120Aas a result of a query related to the content of a page, such as a pagedisplayed to the user A 120A. In the graph 400, the users 120A-N mayhave searched for Query1, Query2, and Query3. When the users 120A-Nsearched for Query1, Ad1, Ad2, and Ad3 may have been displayed. When theusers 120A-N searched for Query2, Ad3, Ad4, and Ad5 may have beendisplayed. When the users 120A-N searched for Query3, Ad3, Ad5, and Ad6may have been displayed.

Once a link between a query and an advertisement is established, thequery may be weighted, or quantified, based on a metric relating to therelationship between the advertisement and the query. For example, thelink may be weighted based on the click through rate of theadvertisement for the particular query. The click through rate for thelink may be only account for the click-throughs attributed to when theadvertisement is displayed as a result of the query represented by thequery node. The click through rate may be calculated over the period oftime T represented by the historical data, such as the previous day,week, month, year, or any other time period. The weights can be seen inthe graph 400 as values on the lines representing the links. The clickthrough rate for a particular advertisement/query pair may be 0.0 ifnone of the users 120A-N clicked on the advertisement.

FIG. 5 is a flowchart illustrating the operations of the system of FIG.3, or other systems for generating query suggestions using a network ofusers and advertisers. At block 505 the graph processor 322 may identifyhistorical data, such as historical user interaction data, historicaluser click data, historical ad display data, historical ad performancedata, or generally any data relating to queries or the display of theresulting advertisements. The graph processor 322 may retrieve thehistorical data directly from the data store 245 or via the serviceprovider 240. The historical data may represent all of the historicaldata for a given time period T, such as the previous day, month, year,or any other determinable time period. At block 510 the graph processor322 may identify all of the individual queries in the historical data.The queries may represent a set of search terms or queries searched forby the users 120A-N during the time period T, such as through a searchengine provided by the service provider 130. There may be more than oneinstance of a query if it was searched for more than once by the users120A-N; however, the underlying data describing the particular userinteraction may differ, and thus each query is processed individually.Alternatively or in addition the queries may represent a set of termsrelated to the content of a page, such as a page displayed to the users120A-N during the time period T. The page may have been served to theusers 120A-N by the service provider 130 and/or a service providerpartner.

At block 515 the graph processor 322 may retrieve the first query, q,from the identified user queries. At block 520 the graph processor 322may identify the advertisements displayed to the user A 120A when theuser A 120A searched for q, or the advertisements displayed to the userA 120A as a result of content relating to q. At block 525 the graphprocessor 322 may retrieve the first advertisement, a, displayed forquery q. At block 530 the graph processor 322 may process the dataassociated with this particular pairing of the query q and theadvertisement a to generate raw context data. The operations ofprocessing the data associated with the query/advertisement pair may bedemonstrated in detail in FIG. 6.

At block 535 the graph processor 322 may determine if the raw contextdata of the query/advertisement pairing is unique. To determine if theraw context data is unique the graph processor 322 may compare the rawcontext data with existing raw context data in the graph data store 326.If the raw context data is unique the system 300 may move to block 545to store the raw context data of the query/advertisement pair. If theraw context data is not unique the system 300 may move to block 550. Theraw context data of query/advertisement pair may be described in FIG. 6.

At block 545 the graph processor 322 may store the raw context datarepresenting the query/ad pair in the graph data store 326. At block 550the graph processor 322 may determine whether there are moreadvertisements which were displayed as a result of the query q. If thereare more advertisements, the system 300 may move to block 555. At block555 the graph processor 322 may retrieve the next advertisement for thequery q. The system 300 may then return to block 530 and repeat theoperations for the advertisement. Once the system 300 has cycled throughall of the advertisements for the query the system 300 may move to block560. At block 560 the graph processor 322 may determine if there areremaining queries in the historical data. If there are no remainingqueries then the system 300 may move to block 370. If there are morequeries in the historical data then the system 300 may move to block565. At block 565 the graph processor 322 may retrieve the next query.The system 300 may the return to block 520 and repeat the operations forthe query.

At block 570 the system 300 may generate a link for each unique query/adpair. The operations of generating the links for the query/ad pairs maybe elaborated in FIG. 7. The query/ad links may be generated at a higherlevel of granularity than the raw context data of a query/ad pair. Thusthere may be one query/ad link representing the raw context data ofseveral query/ad pairs.

FIG. 6 is a flowchart illustrating the operations of identifying the rawcontext data representing a query/advertisement pairing in the system ofFIG. 3, or other systems for generating query suggestions using anetwork of users and advertisers. At block 605 the graph processor 322may identify a query q, such as the first query selected from thehistorical dataset in block 515 of FIG. 5. At block 610 the graphprocessor 322 may identify the advertisements that may have beendisplayed to one of the users 120A-N, such as the user A 120A, after theuser A 120A searched for the query q, or the advertisements displayed toone of the users 120A-N as a result of the content q may have beenrelated to.

At block 615 the graph processor 322 may identify the DUDE state D ofthe query. The DUDE state may refer to the number of advertisements thatmay have been prominently displayed to the user A 120A, such as topadvertisements. Top advertisements are shown in more detail in FIGS. 13and 14 below. Since the DUDE state indicates the number of prominentadvertisements displayed to the user A 120A, the higher the value of theDUDE state the more likely that the user A 120A may have clicked on oneof the advertisements. Therefore the DUDE state may need to be accountedfor in order to accurately determine the effectiveness of theadvertisements of the revenue generators 110A-N. The value of the DUDEstate for the query q may be obtained from the historical data.

At block 620 the graph processor 322 may retrieve the firstadvertisement a displayed for the query q. At block 625 the graphprocessor 322 may determine whether the advertisement a was displayed asa result of a query suggestion from a matching system. As previouslymentioned, the service provider server 240 may implement one or morematching systems that may suggest queries that may relate to the queryof the user A 120A. Advertisements may be retrieved from the ad datastore 318 for the original query of the user A 120A and any queriessuggested by the matching systems. The most relevant ads may bedisplayed to the user A 120A. Alternatively or in addition the ads withthe highest bids, for the original query or any suggested queries, maybe displayed to the user A 120A, or any combination of the bid and therelevance. Data indicating whether the advertisement a was displayed asa result of a query suggested by a matching system may be obtained fromthe historical data.

If the advertisement a was displayed as a result of a query suggestionof a matching system, the system 300 may move to block 630. At block 630the graph processor 322 may identify the query q′ that was suggested bythe matching system. The graph processor 322 may also identify thematching system M that suggested the query q′. The suggested query q′and the matching system M may be obtained from the historical data.Storing the matching system identification M may allow the system 300 toattribute the value of a link to the matching system that generated thesuggestion. If the advertisement a was not displayed as a result of aquery suggestion of a matching system, the system 300 may move to block635.

At block 635 the graph processor 322 may calculate the average rank r ofthe advertisement a when it was displayed as a result of any instance ofthe query q. The rank of an advertisement may be the order in which itwas displayed on the page to the user A 120A. For example, if theadvertisement was the first advertisement displayed it may have a rankof 1, the second ad a rank of 2, and so on. FIGS. 16 and 17 below mayelaborate on the rank of an advertisement. The graph processor 322 maycalculate the sum of the each rank of the advertisement a when it wasdisplayed as a result of any instances of the query q, regardless ofwhether a was displayed due to a matching system. The sum may then bedivided by the number of times the advertisement a was displayed as aresult of the query q to calculate the average rank r. The data for theaverage rank calculation may be obtained from the historical data.

At block 640 the graph processor 322 may calculate the total number ofclicks C, the total number of impressions I and/or the total number ofconversions V for the advertisement a when it was displayed on a resultspage with a DUDE state D as a result of the query q. The graph processor322 may calculate the total number of impressions I by retrieving fromthe historical data the number of times the advertisement a wasdisplayed to the users 120A-N as a result of the query q, on a page witha DUDE state of D, regardless of whether a was displayed because of asuggested query. The graph processor 322 may calculate the total numberof clicks C by retrieving from the historical data the number of timesone of the users 120A-N clicked on the advertisement a on a searchresults page with a DUDE state D after searching for the query q,regardless of whether a was displayed because of to a suggested query.The graph processor 322 may calculate the total number of conversions Vby retrieving from the historical data the number of times one of theusers 120A-N performed a desired action on a web site of one of therevenue generators 110A-N, such as making a purchase, after searchingfor the query q and clicking on the advertisement a on a search resultspage with a DUDE state D, regardless of whether a was displayed due to asuggested query.

At block 650 the graph processor 322 may identify the average cost perclick ppc for the advertisement a when it was retrieved by query qduring the time period T. The graph processor 322 may calculate theaverage cost per click by calculating the sum of the cost for each clickon the advertisement a when it was retrieved by query q in thehistorical data and dividing the sum by the total number of clicks onthe advertisement a when it was retrieved by query q.

At block 655 the graph processor may aggregate the identified rawcontext data relating to the advertisement a and query q pair. The rawcontext data may include C the total clicks on ad listing a for query q,I the total impressions of ad listing a for query q, V the totalconversions attributed to a click on ad listing a for query q, M (ifany) the match type that retrieved ad listing a for query q q′ (if any)the actual bidded term responsible for the display of the ad listing a,D the DUDE state at the time of serving, r the average rank of the ad awhen retrieved by query q, and ppc the average cost the revenuegenerator responsible for advertisement a pays per click when a isretrieved by q. The total number of clicks C, impressions I, andconversions V for a q/a pair may be calculated by taking a summation ofthe individual values for each DUDE state D that may exist for the q/apair.

At block 660 the graph processor 322 may determine whether theaggregated raw context data relating to the q/a pair is unique. Thegraph processor 322 may search the graph data store 326 for an instanceof a query/ad pair with the same raw context data as the q/a pair. Theq/a pair may be unique if no other query/ad pair exists in the graphdata store 326 with the same query q, advertisement a, match type M,suggested query q′, and DUDE state Dr. If no query/ad pair is found inthe graph data store 326 matching the raw context data of the q/a pairthen the q/a pair may be unique. If the q/a pair is unique the system300 may move to block 665. At block 665 the graph processor 322 maystore the raw context q/a data in the graph data store 322. If the q/apair is not unique, the system 300 may move to block 670.

At block 670 the graph processor 322 may determine if there areadditional advertisements which were displayed when the user A 120Asearched for the query q. If additional advertisements exist, the system300 may move to block 675. At block 675 the graph processor 322 mayselect the next advertisement. The system 300 may then return to block625 and repeat the operations for the selected advertisement. The graphprocessor 322 may cycle through the operations for each of theadvertisements displayed to the user A 120A after searching for thequery q.

FIG. 7 is a flow chart illustrating the operations of building a linkbetween each unique query and advertisement in the system of FIG. 3, orother systems for generating query suggestions using a network of usersand advertisers. At block 705 the graph processor 322 may identify allof the query/ad raw context data stored in the graph data store 326. Atblock 710 the graph processor 322 may select the first raw contextquery/ad data, q/a.

At block 715 the graph processor 322 may determine whether a link existsin the graph data store 326 for q and a. A link between a q and a may bereferred to as (q, a). The links may represent the framework for thequery/advertisement graph and may be stored in a separate data structurefrom the query/ad pair data, such as a separate database table. Theremay only be one link for a given q and an a, while there may be severalraw context data entries for a given q and a, such as raw context datawith different match types, suggested queries, and/or DUDE states. Thus,the graph processor 322 may search the graph data store 326 to determineif a link from the query q to the ad listing a exists. If a link doesnot exist, the system 300 may move to block 720.

At block 720 the graph processor 322 may generate a link (q, a) betweenthe query q and the advertisement a. The link may include an associationbetween the query q and the advertisement a, such as a data entrylinking the two. Visually the link may represent an edge in thebipartite graph.

At block 725 the graph processor 322 may calculate the weight of the (q,a) link. The weight may be thought of as the strength of the associationbetween the query q and the advertisement a. The weight may alsorepresent the relevance of the advertisement a to the query q. Theweight may be represented as w(q, a). Since the DUDE state may have animpact on the effectiveness metrics the weights may often be calculatedfor each individual DUDE state D of a query q. The weight of (q, a) fora particular DUDE state D may be represented as w(q, a, D).

The weight, or relevance and/or utility measure, may be represented byseveral different metrics, such as clicked or not clicked, total clicks,un-normalized click through rate, position normalized click throughrate, or generally any metric that may indicate the relevance or utilityof q to a. Some examples of utility may include whether a conversionoccurred or not, total conversions, un-normalized conversion rate, orposition normalized conversion rate. A q may only have one relevancemeasure w(q,a) for any given a; in the position normalized case a q mayhave only one DUDE state during the time period T.

The value of clicked or not clicked weight may be 1 if a was clicked atleast once as a result of q with a DUDE state of D over the period oftime T, or 0 otherwise. The graph processor 322 may determine that a wasclicked at least once if the total clicks, C, for (q, a) is greater than0. The total clicks C may be determined from data stored in the graphdata store 326. A weight of total clicks may be the total number ofclicks for (q, a). The total clicks may be determined from data in thegraph data store 326. A weight of total clicks for a given DUDE statemay be determined by:

w(q,a,D)=Clicks(q,a,D).

A weight of an un-normalized click through rate may be the total clicksC for (q, a) divided by the total number of impressions I for (q, a).The total clicks and total impressions (q, a) may be determined fromdata in the graph data store 326. The weight as an un-normalized clickthrough rate for a particular DUDE state D of (q, a) may be determinedby:

${w\left( {q,a,D} \right)} = {\frac{{Clicks}\left( {q,a,D} \right)}{I\left( {q,a,D} \right)}.}$

A weight of a position normalized click through rate of (q,a) for DUDEstate D, also referred to as the Clicks over Expected Clicks (COEC) maybe determined by:

${w\left( {q,a,D} \right)} = {{C\; O\; E\; {C\left( {q,a,D} \right)}} = \frac{{Clicks}\left( {q,a,D} \right)}{{I\left( {q,a,D} \right)} \cdot {{refCTR}\left( {D,r} \right)}}}$

where r may be the average rank associated with the (q, a). The refCTRmay be a reference click through rate curve for the DUDE state D andaverage rank r of (q, a) averaged over all ads stored in the graph datastore 326. Since the rank r stored in the graph data store 326 is anaverage rank, the average ranks may be rounded to the nearest integer.The average rank for the (q, a) may be retrieved from the graph datastore 326. The refCTR may be calculated by:

${{refCTR}\left( {D,r} \right)} = {\frac{\sum\limits_{a \in A}{C\left( {D,r,a} \right)}}{\sum\limits_{a \in A}{I\left( {D,r,a} \right)}}.}$

Alternatively or in addition, two weights may be calculated, a weightbased on clicks, w1(q,a,D)=Clicks(q,a,D), and a weight based onconversions, w2(q,a,D)=Conversions(q,a,D). The two weights, w1 and w2,may be combined to determine the weight w. The weights may be combinedby the following calculation:

${w\left( {q,a,D} \right)} = {{k*\frac{w\; 1\left( {q,a,D} \right)}{{I\left( {q,a,D} \right)} \cdot {{refCTR}\left( {D,r} \right)}}} + {\left( {1 - k} \right)*w\; 2{\left( {q,a,D} \right).}}}$

In this case k may be a system constant, such as 0.1, 1, 10, or anyvalue.

Alternatively or in addition the weight may be scaled by a factor,referred to as the inverse advertiser frequency (IAF). An individual IAFmay be determined for each advertisement. The IAF may indicate theoverall importance of an advertisement a to a query q as compared toother advertisements. The IAF may be computed from the log data or beassigned through some other definition or heuristic process. Anadvertisement a that is associated with a very large number of queriesmay not provide a strong indication of an association between a and anyone of the queries. In this instance the IAF may be a very small value.If an advertisement is associated with a very small number of queries,the advertisement may be specialized to address solely this narrow setof queries. In this instance the IAF may be a very large value. Afterthe IAF is determined it may be used to adjust, or scale, the weightthrough the following calculation:

w(q,a,D)=w(q,a,D)*IAF.

Alternatively or in addition, the weight may be adjusted to account forthe reliability of the weight. The reliability of a weight may dependupon the number of values, such as clicks, that contribute to theweight. For example, weights derived from only a few clicks may beunreliable. In order to account for the reliability of a weight theweight may be adjusted to incorporate a measure of reliability in itsestimate. In the case of weights derived from clicks, a reliabilityfactor rf may be determined. The rf may be equal to 1.0 if there aremore than 100 clicks, indicating a reliable weight or value. If thereare fewer than 100 clicks the rf may be a value between 0 and 1.0. Asthe number of clicks approaches 0, the rf may also approach 0. In oneinstance the rf may be linearly related to the number of clicks below100. For example, if there are 50 clicks, then the rf may be 0.5, and ifthere are 25 clicks the rf may be 0.25. Once the rf is determined it maybe applied to the weight between q and a by the following calculation:

w(q,a,D)=w(q,a,D)*rf.

Alternatively or in addition the reliability factor rf may also bedetermined from the conversion rate associated with the queryadvertisement pair. A high conversion rate may indicate a strong linkbetween the query q and the advertisement a. Query/ad pairs with higherconversion rates may be considered more significant than those withlower conversion rates. The conversion rate may be used to calculate thereliability factor separate from, or in addition to, using the clicks.For example, a query/ad pair with a low number of clicks may still bereliable if it has a high conversion rate. Once the rf factor due toconversions is determined it may be applied to the following equation toadjust the weight:

w(q,a,D)=w(q,a,D)*rf.

After calculating a weight for the link (q, a) and/or a weight for eachDUDE state D that exists for (q, a), the system 300 may move to block730. At block 730 the graph processor 322 may add the weights to thedata entry representing the link (q, a). At block 735 the graphprocessor 322 may store the data representing the link (q, a), includingthe weights, in the graph data store 326. If the query/ad pair was notunique in block 715, the system 300 may move to block 740.

At block 740 the graph processor 322 may determine whether there areaany additional query/ad pairs. If there are additional query/ad pairsthe system 300 may move to block 745. At block 745 the graph processor322 may select the next query/ad pair. The system 300 may then move toblock 715 and repeat the operations for the query ad/pair. The graphprocessor 322 may repeat the operations for each query/ad pairidentified in the graph data store 326.

FIG. 8 is a flow chart illustrating the operations of using a network ofusers and advertisers built by the system of FIG. 3, or other systemsfor generating query suggestions using a network of users andadvertisers, to suggest queries related to a query q. At block 810 thegraph component 320 may receive a query q, such as from the serviceprovider server 240. The query q may have been searched for by one ofthe users 120A-N, such as the user A 120A. Links between the query q andadvertisements may exist in the graph data store 326. The query q may becommunicated to the graph analyzer 324.

At block 820 the graph analyzer 324 may determine whether the graph datarepresenting the query/advertisement graph was pre-processed. The graphmay be pre-processed offline to build all of the outputs that the graphmay be utilized to generate, such as queries related to a query q. Theoutputs for a given query may be stored in a hashmap to enable a quickand efficient lookup of the data. In very large implementations of thesystem 300 the processing delay may require calculating any potentialoutputs offline. The steps that follow may be performed offline if thegraph data is pre-processed.

If the graph data was not pre-processed the system 300 may move to block830. At block 830 the graph analyzer 324 may identify all queries Q andall advertisements A which are a part of a link in the graph data store326. At block 840 the graph analyzer 324 may calculate a relevance valueR for each query in Q. The relevance may indicate how relevant eachquery in Q is to the query q received from the service provider server240. For a given query q′ in Q, the relevance value R for (q, q′) may bedetermined by the following equation:

${R\left( {q,q^{\prime}} \right)} = {\frac{\sum\limits_{a \in A}{\left( {{w\left( {q,a,D_{q}} \right)} - {\overset{\_}{W}}_{q}} \right) \cdot \left( {{w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} - {\overset{\_}{W}}_{q^{\prime}}} \right)}}{\sqrt{\sum\limits_{a \in A_{q}}\left( {{w\left( {q,a,D_{q}} \right)} - {\overset{\_}{W}}_{q}} \right)^{2}}\sqrt{\sum\limits_{a \in A_{q^{\prime}}}\left( {{w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} - {\overset{\_}{W}}_{q^{\prime}}} \right)^{2}}}.}$

In R(q, q′), Dq may be the DUDE state of (q, a) and Dq′ may be the DUDEstate of (q′, a). The graph analyzer 324 may obtain the weights,w(q,a,D), from the graph data store 326. W _(q) may be the weight valuefor the position normalized click through rate as calculated by:

${w\left( {q,a,D} \right)} = {{{COEC}\left( {q,a,D} \right)} = {\frac{{Clicks}\left( {q,a,D} \right)}{{I\left( {q,a,D} \right)} \cdot {{refCTR}\left( {D,r} \right)}}.}}$

Alternatively or in addition W _(q) may be calculated by:

${\overset{\_}{W}(a)} = {\frac{\sum\limits_{{q \in Q_{a}},D}{{{Clicks}\left( {q,a} \right)} \cdot {{COEC}\left( {q,a,D} \right)}}}{\sum\limits_{{q \in Q_{a}},D}{{Clicks}\left( {q,a} \right)}}.}$

Alternatively or in addition, in some situations, such as when thedistribution scales of the weights are relatively equal, the relevancevalue R of (q, q′) may be determined by:

${R\left( {q,q^{\prime}} \right)} = {\frac{\sum\limits_{a \in A}{\left( {w\left( {q,a,D_{q}} \right)} \right) \cdot \left( {w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} \right)}}{\sqrt{\sum\limits_{a \in A_{q}}\left( {w\left( {q,a,D_{q}} \right)} \right)^{2}}\sqrt{\sum\limits_{a \in A_{q^{\prime}}}\left( {w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} \right)^{2}}}.}$

The value of R(q, q′) may be further enhanced by including an overlapfactor OF. The overlap factor OF may describe the number ofadvertisements the queries q and q′ may have in common and/or the amountof search traffic the queries q and q′ may have in common. For example aquery q may link to 10 advertisements, a query q′ may link to 5advertisements, and the queries q and q′ may share 3 advertisements incommon. The value of R(q, q′) may then be adjusted by the followingcalculation:

R(q,q′)=R(q,q′)*OF

After calculating R(q, q′) for every q′ in Q, the system 300 may move toblock 850. At block 850 the graph analyzer 324 may order the queries inQ in descending order based on their R(q, q′) value. At block 860 thegraph analyzer 324 may select the top N queries with the highest R(q,q′), where N is any number, such as five. Alternatively or in additionthe service provider server 240 may identify the number of queries to beselected.

If the graph analyzer 324 determined that the graph was pre-processed atblock 820, the system 300 may move to 835. If the graph waspre-processed, the steps of blocks 830, 840, 850 and 860 may have beenperformed offline. The N most relevant queries may have been stored in adata structure, such as a hash map, keyed by the query q. The offlineprocessing may have generated a hash map for every q in Q. Thus when aquery q is received, the graph analyzer 324 only needs to perform aquick lookup to identify the queries most relevant to q. At block 835the graph analyzer 324 may perform a lookup to identify the queries mostrelated to the query q.

At block 870 the graph analyzer 324 may communicate the original query qand the N most relevant queries to the ad serving system 310. The adserving system 310 may mark the relevant queries as suggested queriesfrom the query/advertisement network. Thus the match type of thesuggested queries may be the query/advertisement network. The ad servingsystem 310 may serve advertisements which bid on the suggested queriesin addition to those which bid on the query q of the user A 120A. Thesuggested queries may also be communicated to the service provider 240.The service provider 240 may include the suggested queries on the searchresults page. The suggested queries may assist the user A 120A in theirsearch.

FIG. 9 is a flow chart illustrating the operations of using a network ofusers and advertisers built by the system of FIG. 3, or other systemsfor generating query suggestions using a network of users andadvertisers, to determine the advertisements most relevant to a query q.At block 810 the graph component 320 may receive a query q, such as fromthe service provider server 240. The query q may have been searched forby one of the users 120A-N, such as the user A 120A or the query q maybe a set of terms related to content on a page displayed to the user A120A. Links between the query q and advertisements may exist in thegraph data store 326. The query q may be communicated to the graphanalyzer 324.

At block 920 the graph analyzer 324 may determine whether the graph datarepresenting the query/advertisement graph was pre-processed. The graphmay be pre-processed offline to determine all of the outputs that thegraph may be utilized to generate, such as the most relevantadvertisements for a query q. The outputs for a given query may bestored in a hashmap to enable quick and efficient lookup of the data. Invery large implementations of the system 300 the processing time mayrequire calculating outputs offline. The steps that follow may beperformed offline if the graph data is pre-processed.

If the graph data was not pre-processed the system 300 may move to block930. At block 930 the graph analyzer 324 may identify all queries Q andall advertisements A which are a part of a link in the graph data store326. At block 940 the graph analyzer 324 may calculate a relevance valueR for each query q′ in Q. The relevance may indicate how relevant eachquery is to the query q received from the service provider server 240.For a given query q′ in Q, the relevance value R of (q, q′) may bedetermined by:

${R\left( {q,q^{\prime}} \right)} = {\frac{\sum\limits_{a \in A}{\left( {{w\left( {q,a,D_{q}} \right)} - {\overset{\_}{W}}_{q}} \right) \cdot \left( {{w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} - {\overset{\_}{W}}_{q^{\prime}}} \right)}}{\sqrt{\sum\limits_{a \in A_{q}}\left( {{w\left( {q,a,D_{q}} \right)} - {\overset{\_}{W}}_{q}} \right)^{2}}\sqrt{\sum\limits_{a \in A_{q^{\prime}}}\left( {{w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} - {\overset{\_}{W}}_{q^{\prime}}} \right)^{2}}}.}$

In R(q, q′), Dq may be the DUDE state of (q, a) and Dq′ may be the DUDEstate of (q′, a). The graph analyzer 324 may obtain the weights,w(q,a,D), from the graph data store 326. W _(q) may represent the weightvalue for the position normalized click through rate calculated by:

${w\left( {q,a,D} \right)} = {{{COEC}\left( {q,a,D} \right)} = {\frac{{Clicks}\left( {q,a,D} \right)}{{I\left( {q,a,D} \right)} \cdot {{refCTR}\left( {D,r} \right)}}.}}$

Alternatively or in addition W _(q) may be calculated by:

${\overset{\_}{W}(a)} = {\frac{\sum\limits_{{q \in Q_{a}},D}{{{Clicks}\left( {q,a} \right)} \cdot {{COEC}\left( {q,a,D} \right)}}}{\sum\limits_{{q \in Q_{a}},D}{{Clicks}\left( {q,a} \right)}}.}$

Alternatively or in addition, in some situations, such as when thedistribution scales of the weights are relatively equal, the relevancevalue R of (q, q′ may be determined by:

${R\left( {q,q^{\prime}} \right)} = {\frac{\sum\limits_{a \in A}{\left( {w\left( {q,a,D_{q}} \right)} \right) \cdot \left( {w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} \right)}}{\sqrt{\sum\limits_{a \in A_{q}}\left( {w\left( {q,a,D_{q}} \right)} \right)^{2}}\sqrt{\sum\limits_{a \in A_{q^{\prime}}}\left( {w\left( {q^{\prime},a,D_{q^{\prime}}} \right)} \right)^{2}}}.}$

After calculating R(q, q′) for every q′ in Q, the system 300 may move toblock 950. At block 950 the graph analyzer 324 may order the queries inQ in descending order based on their R(q, q′) value. At block 960 thegraph analyzer 324 may select the top K queries with the highest R(q,q′), where K may be any number, such as five.

At block 970 the graph analyzer 324 may use the top K queries,represented by q1, . . . , qK, to calculate a predicted relevancebetween q and each advertisement a existing in A. The graph analyzer mayuse the following formula to calculate the predicted relevance ŵ(q,a)for each a in A:

${\hat{w}\left( {q,a} \right)} = {\frac{\sum\limits_{k = 1}^{K}{{R\left( {q_{k},q} \right)} \cdot {w\left( {q_{k},a} \right)}}}{\sum\limits_{k = 1}^{K}{{R\left( {q_{k},q} \right)}}}.}$

The values for R(q_(k),q) may have previously been calculated at block940. The values for w(q_(k),a) may be stored in the graph data store326.

Alternatively or in addition, the mean of the relevance weightdistribution may be decoupled from the estimation and the mean may beadded back into the final prediction as follows:

${\hat{w}\left( {q^{*},a^{*}} \right)} = {{\overset{\_}{W}}_{q^{*}} + {\frac{\sum\limits_{k = 1}^{K}{{R\left( {q_{k},q^{*}} \right)} \cdot \left( {{w\left( {q_{k},a^{*}} \right)} - {\overset{\_}{W}}_{q_{k}}} \right)}}{\sum\limits_{k = 1}^{K}{{R\left( {q_{k},q^{*}} \right)}}}.}}$

W _(q) may be the weight value for the position normalized click throughrate and may be determined by:

${w\left( {q,a,D} \right)} = {{{COEC}\left( {q,a,D} \right)} = {\frac{{Clicks}\left( {q,a,D} \right)}{{I\left( {q,a,D} \right)} \cdot {{refCTR}\left( {D,r} \right)}}.}}$

Once the predicted relevance ŵ(q,a) has been calculated for each a in A,the system 300 may move to block 980. At block 980 the graph analyzer324 may sort, in descending order, the predicted relevances ŵ(q,a) foreach a in A. At block 985 the top D advertisements most relevant to thequery q may be selected, where D may be any number, such as five.

If at block 920 the graph was pre-processed the system 300 may move to935. If the graph was pre-processed, the steps of blocks 930, 940, 950,960, 970, and 980, and 985 may have been performed offline. The mostrelevant advertisements may have been stored in a data structure, suchas a hash map, keyed by the query q. The offline processing may havegenerated a hash map for every q in Q. Thus when a query q is received,the graph analyzer 324 only needs to perform a quick lookup to identifythe advertisements most relevant to q. At block 935 the graph analyzer324 may perform a lookup on the data structure to identify the top Dadvertisements most relevant to the query q, where D may be any number,such as five.

At block 990 the graph analyzer 324 may communicate the original query qand the advertisements most relevant to q to the service provider server240 and/or the ad serving system 310. The ad serving system 310 may usethe most relevant advertisements to supplement advertisements bid on forthe query q. Alternatively or in addition the graph analyzer 324 maycommunicate the most relevant advertisements directly to the serviceprovider server 240. In this case the service provider server 240 mayonly include the advertisements identified by the graph analyzer 324 inthe search results of the user A 120A and bypass the ad serving system310.

FIG. 10 is a flow chart illustrating the operations of using a networkof users and advertisers built by the system of FIG. 3, or other systemsfor generating query suggestions using a network of users andadvertisers, to determine the value attributed to a suggested query.Determining the value attributed to a suggested query may assist therevenue generators 110A-N in optimizing their advertising campaigns byidentifying the most profitable queries. Alternatively or in addition tothe value attributed to a suggested query may indicate queries relatedto content on a page. The revenue generators 110A-N may then focus theiradvertising campaigns on the most profitable queries. The serviceprovider 130 may provide the revenue generators 110A-N with reportsindicating the effectiveness or value of each of the queries they bidon. The reports may further indicate the value of each query attributedto keyword suggestions and/or matching systems. In some instances therevenue generators 110A-N may not have been bidding on the suggestedkeywords and may modify their advertising campaigns to include bids onthe suggested keywords.

At block 1005 the graph analyzer 324 may receive a query/ad link (q, a)and a suggested query q′ for the link (q, a). The suggested query q′ mayhave been previously suggested for q resulting in the advertisement abeing displayed to one of the users 120A-N. Thus the link (q,a) may havea raw context stored in the graph data store 326 that includes querysuggestion q′ and match type M. The service provider server 240 may havecommunicated (q, a) and q′ to the graph analyzer 324.

At block 1010 the graph analyzer 324 may identify all queries Qidentified in the link data stored on the graph data store 326. Thegraph analyzer 324 may calculate the relevance values R(q, q″) for q andall q″ in Q, such as every query in Q. After calculating the relevancevalues for all the queries, the system 300 may move to block 1015. Atblock 1015 the graph analyzer 324 may sort the queries q″ in descendingorder based on their relevance values R(q, q″). At block 1020 the graphanalyzer 324 may identify the top K queries, where K is any number, suchas five. The graph analyzer 324 may place the top K queries into a set,Q1(q). The steps described in blocks 1010, 1015 and 1020 may bedescribed in more detail in FIGS. 8 and 9 above.

At block 1025 the graph analyzer 324 may create a second build of thegraph without the link (q, a). If neither q nor a are part of any otherlink, then the second build does not have to be performed. At block 1030the graph analyzer 324 may calculate the relevancy values R(q, q″) basedon the second build of the graph for all q″ which may be an element ofQ. At block 1035 the graph analyzer 324 may sort the queries indescending order based on their relevance values R(q, q″). At block 1040the graph analyzer 324 may identify the top K queries from the secondbuild, where K is any number, such as five. The graph analyzer 324 mayplace the top K queries from the second build into a second set, Q2(q).The steps described in blocks 1030, 1035 and 1040 may be described inmore detail in FIGS. 8 and 9 above.

At block 1045 the graph analyzer 324 may determine a set of queriesHQ(q), calculated by Q1(q)−Q2(q). If neither q nor a are part of anyother link, and the second build was not performed, then HQ(q)=Q1(q).

At block 1050 the graph analyzer 324 may calculate the residual value ofthe link {umlaut over (ν)}(q,a), or the summation of the valueattributed to each query in HQ(q). The value of each query in HQ(q) maybe calculated by: ν(q,a)=w(q,a)·ppc(q,a). The weights of w(q,a) may betotal clicks or position normalized CTR. When the value of the weight istotal clicks, the measure of value may be simply a count of the totalaggregated revenue brought by the link over the time period T.Alternatively or in addition the conversion rates may be used instead ofppc. Thus the {umlaut over (ν)}(q,a) for the graph G stored in the graphdata store may be calculated by:

${\overset{¨}{\upsilon}\left( {q,a,G} \right)} = {\sum\limits_{k \in {{HQ}{({q,a})}}}{{\upsilon \left( {k,a} \right)}.}}$

At block 1055 the graph analyzer 324 may add the value of (q,a) to{umlaut over (ν)}(q,a,G) the total value provided by each query in theset HQ(q). The calculation may be represented as:ζ(q′,(q,a),M)=v((q,a))+{umlaut over (v)}((q,a),G), and the result may bethe value attributed to the suggestion q′ for q with match type M.

At block 1060 the graph analyzer 324 may communicate the valueattributed to the suggested query q′ to the service provider server 240.The service provider server 240 may display the value to one of therevenue generators 110A-N along with other data describing theeffectiveness of their advertising campaigns. The revenue generators110A-N may be able to improve their advertising campaign by directlytargeting suggested keywords with high attributed values.

FIG. 11 is a flow chart illustrating the steps of using a network ofusers and advertisers built by the system of FIG. 3, or other systemsfor generating query suggestions using a network of users andadvertisers, to determine the value attributed to a match type and asuggested query. The operations illustrated in the flowchart of FIG. 11may require less computational complexity to determine the valueattributed to a suggested query than the operations illustrated in theflowchart of FIG. 10.

The service provider 130 may be able to determine the values attributedto each of the matching systems implemented in the ad serving system300. The service provider 130 may be able to optimize ad serving byidentifying the best performing matching systems and the worstperforming matching systems. The best performing matching systems may beused more often and the poorly performing matching systems may be slowlyphased out. At block 1110 the graph analyzer 324 may receive a query/adlink (q, a) and a suggested query q′ for the link (q, a). The suggestedquery q′ may have been previously suggested for q, resulting in theadvertisement a being displayed to one of the users 120A-N. Thus thelink (q,a) may have a raw context data stored in the graph data store326 that includes query suggestion q′ and match type M. The serviceprovider server 240 may have communicated (q, a) and q′ to the graphanalyzer 324.

At block 1120 the graph analyzer 324 may determine the match type M ofthe suggested query q′. The match type may be obtained from the rawcontext data of the query/ad link (q, a). At block 1130 the queryanalyzer may calculate the average residual value for the link (q, a)attributed to the match type M. The value attributed to the match type Mmay be calculated by:

${{\zeta (M)} = {\frac{1}{E}{\sum\limits_{({{({q,a})},q^{\prime}})}{\zeta \left( {q^{\prime},\left( {q,a} \right),M} \right)}}}},$

where E is the total number of raw content for (q, a) with match type M.The calculation of ζ(q′,(q,a),M) may be elaborated in more detail in theoperations illustrated in the flowchart of FIG. 10.

At block 1140 the graph analyzer 324 may calculate the value attributedto the suggested query q′ for the link (q, a). The value may becalculated by: ζ(q′,(q,a))=v(q,a)+ζ(M). Details on the calculation ofv(q,a) may be found in the operations of the flowchart illustrated inFIG. 10. At block 1150 the graph analyzer 324 may communicate theaverage residual value attributed to the match type M for the link (q,a)and the value attributed to the keyword suggestion q′ for q with matchtype M to the service provider server 240.

FIG. 12 is a flow chart illustrating the steps of using a network ofusers and advertisers built by the system of FIG. 3, or other systemsfor generating query suggestions using a network of users andadvertisers, to integrate valuable suggestions with experimental orunknown suggestions. The service provider 130 may benefit fromidentifying the best performing suggestions and suggesting them at ahigher rate than lower performing suggestions. Furthermore there may bevaluable queries that are not suggested because they may be new,obscure, or otherwise have not been exposed to a large amount oftraffic. The service provider 130 may benefit from experimenting withthese terms to determine if any of them may be profitable.

The query suggestions stored in the graph data store 326 may beseparated into two sets, one where the value is known and another wherethe value is unknown. A suggestion may have a known value if the valueis non-zero and the suggestion has been exposed to a minimal amount oftraffic, i.e., its impression count exceeds some minimum threshold. Atblock 1210 the graph analyzer 324 may calculate the value provided byeach suggestion in the graph historical dataset. The value of thesuggestions may be calculated via the methods outline in FIGS. 10 and11, namely through the equation: ζ(q′,(q,a))=v(q,a)+ζ(M). Alternativelyor in addition the value may be defined by relevance, or weight, such astotal clicks or conversions.

At block 1220 the graph analyzer 324 may sort the suggestions based ontheir attributed values calculated above. At block 1230, the graphanalyzer 324 may place the suggestions with the top K known values intoan exploit set, where K is any number, such as fifty. At block 1240 thegraph analyzer 324 may place unknown values, such as the next J values,into an explore set. The explore set may include suggestions withunknown values or suggestions with known values without enough trafficexposure. The explore set may allow the system 300 to experiment withpast and new suggestions from any match type. The cardinality of theexplore set may be orders of magnitude larger than that of the exploitset. Suggestions in the explore set may be scored for relevance orvalue, such as by any of the aforementioned methods of valuingsuggestions.

The explore set may be separated into two subsets, a live set and anoffline set. The live set may be the explore suggestions that are intrial, or under experimentation. The offline set may be suggestions thatare not currently being used.

At block 1250 the graph analyzer 324 may communicate the sets of queriesto the service provider server 240, and/or the ad serving system 310.The service provider server 240 and/or the ad serving system may suggestthe suggestions from the exploit set and the explore-live set. Theexploit set may be suggested more frequently than the explore-live setto benefit from the proven value provided by the exploit set.

At block 1260, the graph analyzer 324 may repeat the operations after aninterval of time T has passed. The operations may be continuallyrepeated at the interval of time T. The suggestions may be re-valued andre-sorted. Top valued suggestions with sufficient exposure may comprisethe exploit set. The live explore set may be populated with a new batchof suggestions with unknown values. The continual update of the exploitset and the explore-live set may ensure that the exploit set capturesany seasonal queries or other queries whose value may increase due totemporal externalities.

FIG. 13 illustrates an exemplary page 1300 displaying advertisements.The page 1300 may be served by the service provider 130 to the users120A-N and may be a web page displayed on the Internet. The page 1300may include content 1310, such as a list of search results, which maygenerally be the purpose of the page. The page 1300 may be shown withslots for four advertisements. There may be two top ad slots 1320, 1330and two side ad slots 1340, 1350. The number of ads in the top ad slots1320, 1330 may determine the DUDE state of the query. The serviceprovider 130 may attempt to fill the ad slots 1320, 1330, 1340, 1350with advertisements from the sponsored search server 312, or from thegraph analyzer 324.

FIG. 14 is a screenshot of a page 1400 displaying advertisements to theusers 120A-N served from a search engine marketing system implementing asystem for generating query suggestions using a network of users andadvertisers. The page 1400 may be displayed to one of the users 120A-N,such as the user A 120A, when the user A 120A searches for the term“plasma.” The page 1400 may include a search query 1405, content 1410,query suggestions 1460, top ads 1420, side ads 1430 and a popup ad 1470.The content 1410 may include a search results list 1440 based on thesearch query 1405 submitted by the user A 120A, such as “plasma”. Thesearch results list 1440 may include one or more search results 1450. Asearch result 1450 may include a title link 1452, a URL 1454, adescription 1456 and a rank 1458. The top ads 1420 may include one ormore sponsor listings 1422. The side ads 1430 may include sponsoredlistings. The query suggestions 1460 may represent queries that weresuggested by the query analyzer 324. The queries may represent phrasessimilar to the search query 1405 that users 120A-N searched for. Thequery suggestions 1460 may have been generated by the system of FIG. 3.

The title link 1452 may be a clickable link that may reference a site.If one the users 120A-N, such as the user A 120A, clicks on the titlelink 1452, the user A 120A may be forwarded to the site referred to bythe title link 1452. The site referred to by the title link 1452 may bedescribed in the description 1456. The URL 1454 may represent the URL ofthe site referred to by the link 1452. The rank 1458 may represent theorder of the search result 750 in the search results list 1440.

The top ads 1420 and the side ads 3140 may include any combination ofsponsored listings, banner ads and popup ads. The top ads 1420 and theside ads 1430s may represent advertisements that may have been retrievedfrom the sponsored search server 312, the content match server 316 orthe graph analyzer 324. The number of ads in the top ads 1420 mayindicate the DUDE state of the query. The sponsored listing 1422 and/orthe banner ad 1424 may link the users 120A-N to the web site of arevenue generator, such as the revenue generator A 110A, when the users120A-N click on the banner ad 1424 and/or the sponsored listing 1422.The banner ad 1424 may be constructed from an image (GIF, JPEG, PNG), aJavaScript program or a multimedia object employing technologies such asJava, Shockwave or Flash. The banner ad 1424 may employ animation,video, or sound to maximize presence. The images used in the banner ad1424 may be in a high-aspect ratio shape (i.e. either wide and short, ortall and narrow).

The popup ad 1470 may link the users 120A-N to the web site of a revenuegenerator, such as the revenue generator A 110A, when the users 120A-Nclick on the popup ad 1470. The popup ad 1470 may be constructed from animage (GIF, JPEG, PNG), a JavaScript program or a multimedia objectemploying technologies such as Java, Shockwave or Flash. The popup ad1470 may employ animation, video, or sound to maximize presence. Thepopup ad 1470 may run in the same window as the page, or may open in anew window. The popup ad 1470 may be capable of being closed and/orminimized by clicking on an ‘X’ in the corner of the popup ad 1470.

FIG. 15 illustrates a general computer system 1500, which may representa service provider server 240, a third party server 250, an advertisingservices server 260, a graph component 320, a graph processor 322, agraph analyzer 324 or any of the other computing devices referencedherein. The computer system 1500 may include a set of instructions 1524that may be executed to cause the computer system 1500 to perform anyone or more of the methods or computer based functions disclosed herein.The computer system 1500 may operate as a standalone device or may beconnected, e.g., using a network, to other computer systems orperipheral devices.

In a networked deployment, the computer system may operate in thecapacity of a server or as a client user computer in a server-clientuser network environment, or as a peer computer system in a peer-to-peer(or distributed) network environment. The computer system 1500 may alsobe implemented as or incorporated into various devices, such as apersonal computer (PC), a tablet PC, a set-top box (STB), a personaldigital assistant (PDA), a mobile device, a palmtop computer, a laptopcomputer, a desktop computer, a communications device, a wirelesstelephone, a land-line telephone, a control system, a camera, a scanner,a facsimile machine, a printer, a pager, a personal trusted device, aweb appliance, a network router, switch or bridge, or any other machinecapable of executing a set of instructions 1524 (sequential orotherwise) that specify actions to be taken by that machine. In aparticular embodiment, the computer system 1500 may be implemented usingelectronic devices that provide voice, video or data communication.Further, while a single computer system 1500 may be illustrated, theterm “system” shall also be taken to include any collection of systemsor sub-systems that individually or jointly execute a set, or multiplesets, of instructions to perform one or more computer functions.

As illustrated in FIG. 15, the computer system 1500 may include aprocessor 1502, such as, a central processing unit (CPU), a graphicsprocessing unit (GPU), or both. The processor 1502 may be a component ina variety of systems. For example, the processor 1502 may be part of astandard personal computer or a workstation. The processor 1502 may beone or more general processors, digital signal processors, applicationspecific integrated circuits, field programmable gate arrays, servers,networks, digital circuits, analog circuits, combinations thereof, orother now known or later developed devices for analyzing and processingdata. The processor 1502 may implement a software program, such as codegenerated manually (i.e., programmed).

The computer system 1500 may include a memory 1504 that can communicatevia a bus 1508. The memory 1504 may be a main memory, a static memory,or a dynamic memory. The memory 1504 may include, but may not be limitedto computer readable storage media such as various types of volatile andnon-volatile storage media, including but not limited to random accessmemory, read-only memory, programmable read-only memory, electricallyprogrammable read-only memory, electrically erasable read-only memory,flash memory, magnetic tape or disk, optical media and the like. In onecase, the memory 1504 may include a cache or random access memory forthe processor 1502. Alternatively or in addition, the memory 1504 may beseparate from the processor 1502, such as a cache memory of a processor,the system memory, or other memory. The memory 1504 may be an externalstorage device or database for storing data. Examples may include a harddrive, compact disc (“CD”), digital video disc (“DVD”), memory card,memory stick, floppy disc, universal serial bus (“USB”) memory device,or any other device operative to store data. The memory 1504 may beoperable to store instructions 1524 executable by the processor 1502.The functions, acts or tasks illustrated in the figures or describedherein may be performed by the programmed processor 1502 executing theinstructions 1524 stored in the memory 1504. The functions, acts ortasks may be independent of the particular type of instructions set,storage media, processor or processing strategy and may be performed bysoftware, hardware, integrated circuits, firm-ware, micro-code and thelike, operating alone or in combination. Likewise, processing strategiesmay include multiprocessing, multitasking, parallel processing and thelike.

The computer system 1500 may further include a display 1514, such as aliquid crystal display (LCD), an organic light emitting diode (OLED), aflat panel display, a solid state display, a cathode ray tube (CRT), aprojector, a printer or other now known or later developed displaydevice for outputting determined information. The display 1514 may actas an interface for the user to see the functioning of the processor1502, or specifically as an interface with the software stored in thememory 1504 or in the drive unit 1506.

Additionally, the computer system 1500 may include an input device 1512configured to allow a user to interact with any of the components ofsystem 1500. The input device 1512 may be a number pad, a keyboard, or acursor control device, such as a mouse, or a joystick, touch screendisplay, remote control or any other device operative to interact withthe system 1500.

The computer system 1500 may also include a disk or optical drive unit1506. The disk drive unit 1506 may include a computer-readable medium1522 in which one or more sets of instructions 1524, e.g. software, canbe embedded. Further, the instructions 1524 may perform one or more ofthe methods or logic as described herein. The instructions 1524 mayreside completely, or at least partially, within the memory 1504 and/orwithin the processor 1502 during execution by the computer system 1500.The memory 1504 and the processor 1502 also may includecomputer-readable media as discussed above.

The present disclosure contemplates a computer-readable medium 1522 thatincludes instructions 1524 or receives and executes instructions 1524responsive to a propagated signal; so that a device connected to anetwork 235 may communicate voice, video, audio, images or any otherdata over the network 235. Further, the instructions 1524 may betransmitted or received over the network 235 via a communicationinterface 1518. The communication interface 1518 may be a part of theprocessor 1502 or may be a separate component. The communicationinterface 1518 may be created in software or may be a physicalconnection in hardware. The communication interface 1518 may beconfigured to connect with a network 235, external media, the display1514, or any other components in system 1500, or combinations thereof.The connection with the network 235 may be a physical connection, suchas a wired Ethernet connection or may be established wirelessly asdiscussed below. Likewise, the additional connections with othercomponents of the system 1500 may be physical connections or may beestablished wirelessly. In the case of a service provider server 240, athird party server 250, an advertising services server 260, the serversmay communicate with users 120A-N and the revenue generators 110A-Nthrough the communication interface 1518.

The network 235 may include wired networks, wireless networks, orcombinations thereof. The wireless network may be a cellular telephonenetwork, an 802.11, 802.16, 802.20, or WiMax network. Further, thenetwork 235 may be a public network, such as the Internet, a privatenetwork, such as an intranet, or combinations thereof, and may utilize avariety of networking protocols now available or later developedincluding, but not limited to TCP/IP based networking protocols.

The computer-readable medium 1522 may be a single medium, or thecomputer-readable medium 1522 may be a single medium or multiple media,such as a centralized or distributed database, and/or associated cachesand servers that store one or more sets of instructions. The term“computer-readable medium” may also include any medium that may becapable of storing, encoding or carrying a set of instructions forexecution by a processor or that may cause a computer system to performany one or more of the methods or operations disclosed herein.

The computer-readable medium 1522 may include a solid-state memory suchas a memory card or other package that houses one or more non-volatileread-only memories. The computer-readable medium 1522 also may be arandom access memory or other volatile re-writable memory. Additionally,the computer-readable medium 1522 may include a magneto-optical oroptical medium, such as a disk or tapes or other storage device tocapture carrier wave signals such as a signal communicated over atransmission medium. A digital file attachment to an e-mail or otherself-contained information archive or set of archives may be considereda distribution medium that may be a tangible storage medium.Accordingly, the disclosure may be considered to include any one or moreof a computer-readable medium or a distribution medium and otherequivalents and successor media, in which data or instructions may bestored.

Alternatively or in addition, dedicated hardware implementations, suchas application specific integrated circuits, programmable logic arraysand other hardware devices, may be constructed to implement one or moreof the methods described herein. Applications that may include theapparatus and systems of various embodiments may broadly include avariety of electronic and computer systems. One or more embodimentsdescribed herein may implement functions using two or more specificinterconnected hardware modules or devices with related control and datasignals that may be communicated between and through the modules, or asportions of an application-specific integrated circuit. Accordingly, thepresent system may encompass software, firmware, and hardwareimplementations.

The methods described herein may be implemented by software programsexecutable by a computer system. Further, implementations may includedistributed processing, component/object distributed processing, andparallel processing. Alternatively or in addition, virtual computersystem processing maybe constructed to implement one or more of themethods or functionality as described herein.

Although components and functions are described that may be implementedin particular embodiments with reference to particular standards andprotocols, the components and functions are not limited to suchstandards and protocols. For example, standards for Internet and otherpacket switched network transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP)represent examples of the state of the art. Such standards areperiodically superseded by faster or more efficient equivalents havingessentially the same functions. Accordingly, replacement standards andprotocols having the same or similar functions as those disclosed hereinare considered equivalents thereof.

The illustrations described herein are intended to provide a generalunderstanding of the structure of various embodiments. The illustrationsare not intended to serve as a complete description of all of theelements and features of apparatus, processors, and systems that utilizethe structures or methods described herein. Many other embodiments maybe apparent to those of skill in the art upon reviewing the disclosure.Other embodiments may be utilized and derived from the disclosure, suchthat structural and logical substitutions and changes may be madewithout departing from the scope of the disclosure. Additionally, theillustrations are merely representational and may not be drawn to scale.Certain proportions within the illustrations may be exaggerated, whileother proportions may be minimized. Accordingly, the disclosure and thefigures are to be regarded as illustrative rather than restrictive.

Although specific embodiments have been illustrated and describedherein, it should be appreciated that any subsequent arrangementdesigned to achieve the same or similar purpose may be substituted forthe specific embodiments shown. This disclosure is intended to cover anyand all subsequent adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, may be apparent to those of skill in theart upon reviewing the description.

The Abstract is provided with the understanding that it will not be usedto interpret or limit the scope or meaning of the claims. In addition,in the foregoing Detailed Description, various features may be groupedtogether or described in a single embodiment for the purpose ofstreamlining the disclosure. This disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter may be directed toless than all of the features of any of the disclosed embodiments. Thus,the following claims are incorporated into the Detailed Description,with each claim standing on its own as defining separately claimedsubject matter.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other embodiments, which fall withinthe true spirit and scope of the description. Thus, to the maximumextent allowed by law, the scope is to be determined by the broadestpermissible interpretation of the following claims and theirequivalents, and shall not be restricted or limited by the foregoingdetailed description.

1. A method for determining quality of query suggestions using a networkof users and advertisers, the method comprising: generating, from ahistorical dataset, a data representing a network comprising a pluralityof query items representing queries linked to a plurality ofadvertisement items representing advertisements via a plurality ofquery-advertisement link items, wherein each query-advertisement linkitem comprises a weight representing the strength of the associationbetween each linked query item and each linked advertisement item, anadvertisement value representing a value provided by the linkedadvertisement item, a suggested query data describing a query suggestedby a query suggestion system for the query item, and a match typerepresenting the query suggestion system; identifying aquery-advertisement link item in the data representing the networkwherein the query-advertisement link item comprises a suggested querydata; calculating a residual value for the query suggestion systemrepresented by the match type of the query-advertisement link item,wherein the calculation is based on the weight and the advertisementvalue of each query-advertisement link sharing a match type and asuggested query data with the identified query-advertisement link item;calculating a query-advertisement link value for the query advertisementlink based on the weight and the advertisement value of thequery-advertisement link item; determining a query suggestion value ofthe query suggestion described by the suggested query data by adding theresidual value to the query-advertisement link value; and storing thequery suggestion value in a data store.
 2. The method of claim 1 whereineach advertisement item in the plurality of advertisement itemscomprises an online advertisement of an advertiser.
 3. The method ofclaim 2 further comprising linking an advertisement item to a queryitem, via a query-advertisement link item, when the historical datasetindicates a user clicked on the advertisement item after searching forthe query item.
 4. The method of claim 3 further comprising calculatinga weight of each query-advertisement link item based on a total numberof click throughs attributable to each query-advertisement link item inthe historical dataset.
 5. The method of claim 4 wherein the totalnumber of click throughs represents the number of times a plurality ofusers clicked on the advertisement item associated with thequery-advertisement link item after searching for the query itemassociated with the query-advertisement link item.
 6. The method ofclaim 5 wherein the advertisement value of each query-advertisement linkrepresents an amount paid by the advertiser for each click by theplurality of users on the online advertisement.
 7. The method of claim 1further comprising calculating the query-advertisement link value bymultiplying the advertisement value by the weight.
 8. The method ofclaim 1 further comprising communicating the query suggestion value toan advertiser.
 9. A method for generating a report displaying the valueof a query and the value of all queries suggested for the query, themethod comprising: generating, from a historical dataset, a datarepresenting a network comprising a plurality of query itemsrepresenting queries linked to a plurality of advertisement itemsrepresenting advertisements via a plurality of query-advertisement linkitems, wherein each query-advertisement link item comprises a weightrepresenting the strength of the association between each linked queryitem and each linked advertisement item, an advertisement valuerepresenting a value provided by the linked advertisement item, asuggested query data describing a query suggested by a query suggestionsystem for the query item, and a match type representing the querysuggestion system; receiving a query-advertisement link item; generatinga set of distinct suggested query data, wherein the set comprises thesuggested query data of each query-advertisement link item linking thequery item and advertisement item linked by the receivedquery-advertisement link item; calculating a query-advertisement linkvalue for the query advertisement link based on the weight and theadvertisement value of the query-advertisement link item; calculating aquery suggestion value for each query suggestion described by asuggested query data in the set; generating a report, wherein the reportcomprises the query-advertisement link value, and the query suggestionvalue of each distinct query suggestion; and communicating the report toan advertiser.
 10. The method of claim 9 further comprising linking anadvertisement item to a query item when the historical dataset indicatesa user clicked on the advertisement item after searching for the queryitem.
 11. The method of claim 9 wherein the weight of eachquery-advertisement link item in the plurality of query-advertisementlink items comprises a total number of click throughs attributable toeach query-advertisement link item.
 12. The method of claim 11 whereinthe advertisement value of each query-advertisement link item representsan amount paid by the advertiser for each click on the linkedadvertisement item.
 13. The method of claim 9 further comprisingcalculating the query-advertisement link value by multiplying theadvertisement value of the query-advertisement link by the weight of thequery-advertisement link.
 14. The method of claim 9 whereincommunicating the report to an advertiser further comprises: adding thereport to a web page; and serving the web page to the advertiser.
 15. Asystem for determining quality of query suggestions using a network ofusers and advertisers, the system comprising: a memory to store ahistorical dataset, a data representing a network comprising a pluralityof query items representing queries linked to a plurality ofadvertisement items representing advertisements via a plurality ofquery-advertisement link items, wherein each query-advertisement linkitem comprises a weight representing the strength of the associationbetween each linked query item and each linked advertisement item, anadvertisement value representing a value provided by the linkedadvertisement item, a suggested query data describing a query suggestedby a query suggestion system for the query item, and a match typerepresenting the query suggestion system, a residual value, aquery-advertisement link value, and a query suggestion value; and aprocessor operatively connected to the memory, the processor operativeto retrieve the historical dataset, generate the data representing thenetwork, identify a query-advertisement link item in the datarepresenting the network, wherein the query-advertisement link itemcomprises a suggested query data, calculate the residual value of thequery suggestion system represented by the match type of thequery-advertisement link item, wherein the calculation is based on theweight and the advertisement value of each query-advertisement linksharing a match type and a suggested query data with the identifiedquery-advertisement link item, calculate the query-advertisement linkvalue for the query-advertisement link based on the weight and theadvertisement value of the query-advertisement link, determine a querysuggestion value by adding the residual value to the query-advertisementlink value; and store the query suggestion value in the memory.
 16. Thesystem of claim 15 further comprising an interface operatively connectedto the memory and the processor, the interface for communicating with auser.
 17. The system of claim 16 wherein the processor receives thequery-advertisement link item, via the interface, from the user.
 18. Thesystem of claim 15 wherein the residual value represents a valueattributable to the query suggestion system for the suggested query. 19.The system of claim 15 wherein each query item in the plurality of queryitems comprises a search query performed by at least one user in aplurality of users.
 20. The system of claim 15 wherein eachadvertisement item in the plurality of advertisement items comprises anonline advertisement of an advertiser.
 21. The system of claim 15wherein the processor calculates the query-advertisement link value bymultiplying the weight of the query-advertisement link and theadvertisement value of the query-advertisement link.